BLUE21neo: [CentOS6][SOS JobScheduler] 同期の使用例

Create: 2013/03/27
LastUpdate: 2013/05/20

[ メニューに戻る ]

下図の環境を使用します。
環境の詳細を知りたい場合は、メニューに戻って構築手順を参照してください。

ここでは、JobSchedulerの内部ＡＰＩと同期の機能を利用して下図のジョブフローを実現してみます。
手順については、「JobScheduler FAQ」を参考にしました。

このジョブフローを、JobSchduler で、下図のように定義します。

ParallelSample ・・・メイン処理のJobChain。これを実行すると全てのジョブが実行される。
ParallelExecution　・・・　executer ジョブから内部ＡＰＩを使用して実行されるJobChain。２パターンの流れの state を定義

今回、JOEは使用せず、直接ＸＭＬファイルを作成しています。
ジョブ関連の定義と格納するホットフォルダは以下のとおり

/home/jobs/sos-berlin.com/jobscheduler/scheduler/config/live/test/event2

/home/jobs/sos-berlin.com/jobscheduler/scheduler/config/live/test/event2
├── ParallelExecution.job_chain.xml
├── ParallelSample.job_chain.xml
├── executer.job.xml
├── job1.job.xml
├── job2.job.xml
├── job3.job.xml
├── jobA.job.xml
├── jobB.job.xml
└── sync.job.xml

１．sync の定義

同期ジョブを作成します。ファイル名は以下のとおり。

sync.job.xml

内容は以下のとおり。
パラメータで同期に必要なOrder数を指定します。
赤字部分は、JobChainの名称です。パラメータの詳細は、JOCでこのジョブを右クリックし、メニューから[ShowDescprition]を選択してドキュメントを参照してください。

<?xml version="1.0" encoding="ISO-8859-1"?>

<job order="yes"
     tasks="1">
    <description>
        <include file="jobs/JobSchedulerSynchronizeJobChains.xml"/>
    </description>
    <params>
        <param name="ParallelExecution_required_orders"
               value="2"/>
        <param name="ParallelSample_required_orders"
               value="1"/>
    </params>
    <script language="java"
            java_class="sos.scheduler.job.JobSchedulerSynchronizeJobChains"
            java_class_path=""/>
    <run_time/>
</job>

２．ParallelExecution の定義

execution ジョブ（後述）から実行される JobCahin を定義します。ファイル名は以下のとおり。

ParallelExecution.job_chain.xml

内容は以下のとおり。"p1"→"p1.b"→"sync"と、"p2"→"sync"の２つの実行パターンを定義します。
Order生成時に "p1"または"p2"のどちらから実行するかを設定します。

<?xml version="1.0" encoding="ISO-8859-1"?>

<job_chain orders_recoverable="yes">
   <job_chain_node state="p1"
                   job="job1"
                   next_state="p1.b"
                   error_state="error"/>
   <job_chain_node state="p1.b"
                   job="job2"
                   next_state="sync"
                   error_state="error"/>
   <job_chain_node state="p2"
                   job="job3"
                   next_state="sync"
                   error_state="error"/>
   <job_chain_node state="sync"
                   job="sync"
                   next_state="success"
                   error_state="error"/>
   <job_chain_node state="success"/>
   <job_chain_node state="error"/>
</job_chain>

job1、job2、job3 は以下のように定義します。
テスト時にJOCの画面を確認しやすいように10秒スリープします。

<?xml version="1.0" encoding="ISO-8859-1"?>

<job order="yes"
     stop_on_error="no">
    <script language="shell">
        <![CDATA[
echo "$SCHEDULER_JOB_NAME processed $SCHEDULER_ORDER_ID"
echo "Current Timestamp: `date '+%Y-%m-%d %H:%M:%S'`"
echo "Current Hostname: `uname -n`"
echo "Current Username: `whoami`"
sleep 10
exit 0
        ]]>
    </script>
    <run_time/>
</job>

３．ParallelSample の定義

メインのJobChainを定義します。ファイル名は以下のとおり。

ParallelSample.job_chain.xml

内容は、以下のとおり。
executer の state （赤字）は ParallelExecute の開始 state として使用されます。

<?xml version="1.0" encoding="ISO-8859-1"?>

<job_chain orders_recoverable="yes"
           visible="yes">
    <job_chain_node state="100"
                    job="jobA"
                    next_state="p1"
                    error_state="error"/>
    <job_chain_node state="p1"
                    job="executer"
                    next_state="p2"
                    error_state="error"/>
    <job_chain_node state="p2"
                    job="executer"
                    next_state="sync"
                    error_state="sync"/>
    <job_chain_node state="sync"
                    job="sync"
                    next_state="200"
                    error_state="error"/>
    <job_chain_node state="200"
                    job="jobB"
                    next_state="success"
                    error_state="error"/>
    <job_chain_node state="success"/>
    <job_chain_node state="error"/>
</job_chain>

executer ジョブは以下のとおり。
パラメータで指定された JobChain から Order を生成します。
また、job_chain_node の state に設定された値（上記の赤字）を、Order の開始 state として設定します。
なお、このジョブは、Order を生成＆実行するだけで、Orderの実行が完了するまで待機することはありません。

<?xml version="1.0" encoding="ISO-8859-1"?>

<job order="yes"
     stop_on_error="no"
     tasks="3">
    <params>
        <param name="exe_jobchain"
               value="/test/event2/ParallelExecution"/>
    </params>
    <script language="javascript">
        <![CDATA[
function spooler_process(){
   var thisOrder = spooler_task.order;
   var order = spooler.create_order();
   var exe_jobchain = spooler_task.params.value("exe_jobchain");
   spooler_log.info("** exe_jobchain:" + exe_jobchain);

   order.state = thisOrder.state;
   spooler_log.info("** exe_state:" + order.state);

   order.params.merge(thisOrder.params);

   order.id = thisOrder.id + "_" + thisOrder.state;
   spooler_log.info("** Order addes:" + order.id);
   spooler.job_chain( exe_jobchain ).add_order( order );
   return true;
}
        ]]>
    </script>
    <monitor name="configuration_monitor"
             ordering="0">
        <script java_class="sos.scheduler.managed.configuration.ConfigurationOrderMonitor"
                language="java"
                java_class_path=""/>
    </monitor>
    <run_time/>
</job>

jobA、jobBの内容は以下のとおり。

<?xml version="1.0" encoding="ISO-8859-1"?>

<job order="yes"
     stop_on_error="no">
    <script language="shell">
        <![CDATA[
echo "$SCHEDULER_JOB_NAME processed $SCHEDULER_ORDER_ID"
echo "Current Timestamp: `date '+%Y-%m-%d %H:%M:%S'`"
echo "Current Hostname: `uname -n`"
echo "Current Username: `whoami`"
exit 0
        ]]>
    </script>
    <run_time/>
</job>

４．動作確認

JOC で ParallelSample を実行します。
今回は、上で紹介した図に合わせるために、Order ID に "123" を設定します。

下図は、実行中の画面です。
ParallelSample は、jobBの直前で中断しています。
PrallelExecution は、"p1"から開始した Order と、"p2" から開始した Order が実行中です。
PrallelExecution の２つのOrder が sync ジョブまで実行したら、jobB が実行されて正常終了します。

下図は、正常終了後に、実行時間を確認したところです。
PrallelExecution のOrder ID である "123_p1" と "123_p2" は、開始時間が同じなのでパラレルに実行していたことがわかります。

ParallelSample　の実行結果は以下のとおり。

2013-03-27 16:43:45.633 [info]   (Task test/event2/jobA:4921) SCHEDULER-842  Task is going to process Order test/event2/ParallelSample:123, state=100, on Scheduler http://centos6:4444
2013-03-27 16:43:45.634 [info]   (Task test/event2/jobA:4921) 
2013-03-27 16:43:45.634 [info]   (Task test/event2/jobA:4921) Task test/event2/jobA:4921 - Protocol starts in /home/jobs/sos-berlin.com/jobscheduler/scheduler/logs/task.test,event2,jobA.log
2013-03-27 16:43:45.635 [info]   (Task test/event2/jobA:4921) SCHEDULER-918  state=starting (at=never)
2013-03-27 16:43:45.636 [info]   (Task test/event2/jobA:4921) SCHEDULER-987  Starting process: '/bin/sh' '-c' '"/tmp/jobs/sos.lPLnc3"'
2013-03-27 16:43:45.725 [info]   (Task test/event2/jobA:4921) jobA processed 123
2013-03-27 16:43:45.725 [info]   (Task test/event2/jobA:4921) Current Timestamp: 2013-03-27 16:43:45
2013-03-27 16:43:45.725 [info]   (Task test/event2/jobA:4921) Current Hostname: centos6
2013-03-27 16:43:45.725 [info]   (Task test/event2/jobA:4921) Current Username: jobs
2013-03-27 16:43:45.726 [info]   (Task test/event2/jobA:4921) SCHEDULER-915  Process event
2013-03-27 16:43:45.727 [info]   (Task test/event2/jobA:4921) SCHEDULER-843  Task has ended processing of Order test/event2/ParallelSample:123, state=100, on Scheduler http://centos6:4444
2013-03-27 16:43:45.727 [info]   set_state p1, Job /test/event2/executer
2013-03-27 16:43:45.764 [info]   (Task test/event2/executer:4922) SCHEDULER-842  Task is going to process Order test/event2/ParallelSample:123, state=p1, on Scheduler http://centos6:4444
2013-03-27 16:43:45.765 [info]   (Task test/event2/executer:4922) 
2013-03-27 16:43:45.765 [info]   (Task test/event2/executer:4922) Task test/event2/executer:4922 - Protocol starts in /home/jobs/sos-berlin.com/jobscheduler/scheduler/logs/task.test,event2,executer.4922.log
2013-03-27 16:43:45.794 [info]   (Task test/event2/executer:4922) SCHEDULER-918  state=starting (at=never)
2013-03-27 16:43:46.220 [info]   (Task test/event2/executer:4922) Configuration File: /home/jobs/sos-berlin.com/jobscheduler/scheduler/config/live/test/event2/ParallelSample.config.xml not found (Probably running on an agent).
2013-03-27 16:43:46.220 [info]   (Task test/event2/executer:4922) Reading configuration from xml payload...
2013-03-27 16:43:46.233 [WARN]   (Task test/event2/executer:4922) ConfigurationBaseMonitor: Monitor: error occurred preparing configuration: Premature end of file.
2013-03-27 16:43:46.238 [info]   (Task test/event2/executer:4922) ** exe_jobchain:/test/event2/ParallelExecution
2013-03-27 16:43:46.239 [info]   (Task test/event2/executer:4922) ** exe_state:p1
2013-03-27 16:43:46.240 [info]   (Task test/event2/executer:4922) ** Order addes:123_p1
2013-03-27 16:43:46.282 [info]   (Task test/event2/executer:4922) SCHEDULER-843  Task has ended processing of Order test/event2/ParallelSample:123, state=p1, on Scheduler http://centos6:4444
2013-03-27 16:43:46.283 [info]   set_state p2, Job /test/event2/executer
2013-03-27 16:43:46.297 [info]   (Task test/event2/executer:4924) SCHEDULER-842  Task is going to process Order test/event2/ParallelSample:123, state=p2, on Scheduler http://centos6:4444
2013-03-27 16:43:46.297 [info]   (Task test/event2/executer:4924) 
2013-03-27 16:43:46.297 [info]   (Task test/event2/executer:4924) Task test/event2/executer:4924 - Protocol starts in /home/jobs/sos-berlin.com/jobscheduler/scheduler/logs/task.test,event2,executer.4924.log
2013-03-27 16:43:46.332 [info]   (Task test/event2/executer:4924) SCHEDULER-918  state=starting (at=never)
2013-03-27 16:43:46.740 [info]   (Task test/event2/executer:4924) Configuration File: /home/jobs/sos-berlin.com/jobscheduler/scheduler/config/live/test/event2/ParallelSample.config.xml not found (Probably running on an agent).
2013-03-27 16:43:46.740 [info]   (Task test/event2/executer:4924) Reading configuration from xml payload...
2013-03-27 16:43:46.754 [WARN]   (Task test/event2/executer:4924) ConfigurationBaseMonitor: Monitor: error occurred preparing configuration: Premature end of file.
2013-03-27 16:43:46.759 [info]   (Task test/event2/executer:4924) ** exe_jobchain:/test/event2/ParallelExecution
2013-03-27 16:43:46.760 [info]   (Task test/event2/executer:4924) ** exe_state:p2
2013-03-27 16:43:46.761 [info]   (Task test/event2/executer:4924) ** Order addes:123_p2
2013-03-27 16:43:46.803 [info]   (Task test/event2/executer:4924) SCHEDULER-843  Task has ended processing of Order test/event2/ParallelSample:123, state=p2, on Scheduler http://centos6:4444
2013-03-27 16:43:46.803 [info]   set_state sync, Job /test/event2/sync
2013-03-27 16:43:46.818 [info]   (Task test/event2/sync:4926) SCHEDULER-842  Task is going to process Order test/event2/ParallelSample:123, state=sync, on Scheduler http://centos6:4444
2013-03-27 16:43:46.819 [info]   (Task test/event2/sync:4926) 
2013-03-27 16:43:46.819 [info]   (Task test/event2/sync:4926) Task test/event2/sync:4926 - Protocol starts in /home/jobs/sos-berlin.com/jobscheduler/scheduler/logs/task.test,event2,sync.log
2013-03-27 16:43:46.853 [info]   (Task test/event2/sync:4926) SCHEDULER-918  state=starting (at=never)
2013-03-27 16:43:47.348 [info]   (Task test/event2/sync:4926) synchronizing order [123, ] of job chain [/test/event2/ParallelSample]
2013-03-27 16:43:47.362 [info]   (Task test/event2/sync:4926) job chain [/test/event2/ParallelExecution], state [sync] requires 2 orders being present, 0 orders have been enqueued
2013-03-27 16:43:47.366 [info]   (Task test/event2/sync:4926) job chain [/test/event2/ParallelSample], state [sync] requires 1 orders being present, 1 orders have been enqueued
2013-03-27 16:43:47.367 [info]   (Task test/event2/sync:4926) Synchits:0
2013-03-27 16:43:47.368 [info]   SCHEDULER-991  Order has been suspended
2013-03-27 16:43:47.369 [info]   (Task test/event2/sync:4926) order is set on hold: 123, 
2013-03-27 16:43:47.370 [info]   (Task test/event2/sync:4926) synchronized job chain: /test/event2/ParallelSample, state [sync], required orders: 1, pending orders: 1
2013-03-27 16:43:47.370 [info]   (Task test/event2/sync:4926) synchronized job chain: /test/event2/ParallelExecution, state [sync], required orders: 2, pending orders: 0
2013-03-27 16:43:47.371 [info]   (Task test/event2/sync:4926) SCHEDULER-843  Task has ended processing of Order test/event2/ParallelSample:123, state=sync, on Scheduler http://centos6:4444
2013-03-27 16:44:07.127 [info]   set_state 200, Job /test/event2/jobB, suspended
2013-03-27 16:44:07.127 [info]   SCHEDULER-992  Order ist not longer suspended, next start now
2013-03-27 16:44:07.148 [info]   (Task test/event2/jobB:4930) SCHEDULER-842  Task is going to process Order test/event2/ParallelSample:123, state=200, on Scheduler http://centos6:4444
2013-03-27 16:44:07.148 [info]   (Task test/event2/jobB:4930) 
2013-03-27 16:44:07.148 [info]   (Task test/event2/jobB:4930) Task test/event2/jobB:4930 - Protocol starts in /home/jobs/sos-berlin.com/jobscheduler/scheduler/logs/task.test,event2,jobB.log
2013-03-27 16:44:07.149 [info]   (Task test/event2/jobB:4930) SCHEDULER-918  state=starting (at=never)
2013-03-27 16:44:07.149 [info]   (Task test/event2/jobB:4930) SCHEDULER-987  Starting process: '/bin/sh' '-c' '"/tmp/jobs/sos.UaCdoL"'
2013-03-27 16:44:07.223 [info]   (Task test/event2/jobB:4930) jobB processed 123
2013-03-27 16:44:07.223 [info]   (Task test/event2/jobB:4930) Current Timestamp: 2013-03-27 16:44:07
2013-03-27 16:44:07.223 [info]   (Task test/event2/jobB:4930) Current Hostname: centos6
2013-03-27 16:44:07.223 [info]   (Task test/event2/jobB:4930) Current Username: jobs
2013-03-27 16:44:07.223 [info]   (Task test/event2/jobB:4930) SCHEDULER-915  Process event
2013-03-27 16:44:07.224 [info]   (Task test/event2/jobB:4930) SCHEDULER-843  Task has ended processing of Order test/event2/ParallelSample:123, state=200, on Scheduler http://centos6:4444
2013-03-27 16:44:07.224 [info]   set_state success
2013-03-27 16:44:07.224 [info]   SCHEDULER-945  No further job in job chain - order has been carried out
2013-03-27 16:44:07.224 [info]   SCHEDULER-940  Removing order from job chain

[CentOS6][SOS JobScheduler] 同期の使用例 - パラレル実行

１．sync の定義

２．ParallelExecution の定義

３．ParallelSample の定義

４．動作確認