Nightly batch processes????

Your rating: None Average: 2.7 (3 votes)

Guys.... In this post I'm just sharing my thoughts on nightly batch processes rather than writing any technical solution or design.

The other day we had some issues with a few of the batch processes running in our production. Some of the custom mass update processes where running for hours which slowed down the system and in turn crashing one of the app servers. Luckily we have failover app servers configured, so we didn't receive hundreds of emails and tickets logged by the end users. :-)

The mass update program was a very simple application engine program which accepts input data from a flat file and update the Job data of the employees using Component Interface. On that particular day, the input file contained more than 5000 employee rows and the process was running for more than 3 hours.

We (me, our ps admin and some other folks) were discussing about this issue and one of my fellow developer pointed out that we shouldn't be running the mass update processes in the day time and should be scheduled to run in the night (which we call as nightly batch processes) so that the process is run when no users are logged into the system. Our ps admin asked him What he would do if the application is rolled out to more and more countries around the globe and there is no so called "night time" available to run the batch processes i.e. at any point in time some users will be logged into the system and we won't have a bandwidht for the "nightly batch processes".

What would be the solution for the mass updates in such scenarios?
Well, I don't know what are the ideal solutions. Some of the solutions that I can think of right now is data caching and intelligent coding which includes catching the errors, committing the updates in regular intervals so that if the the program is abended due to an error in any particular transaction does not roll back several thousands of perfect transactions which are already completed, logging the committed updates, logging the errors.

Aaaah.. This came out of my frustration with dirty mass updates.. :-)

Folks, let me know your thoughts as well.

Please try to help out with unanswered topics on the forum. Chances are you have had the same issue/question some time in your IT career!

Comments

Have a question? Please ask it on the forum instead.
CompShack's picture
User offline. Last seen 5 years 3 weeks ago. Offline
Joined: 12/09/2007
Posts: 167
Re: Nightly batch processes????

Nice post, thank you for that.

my fellow developer pointed out that we shouldn't be running the mass update processes in the day time and should be scheduled to run in the night

I agree, all of our "demanding" processes such as pay cycle and journal edit and post run after 7 pm. I would suggest that you identify when is your company's peek usage time and avoid running processes then.

I wouldn't worry much about going global and different time zones. I work for a very large company that have offices all over the world and we still run our batch jobs at night. Again, it is always better to run them when traffic is at its minimum.

Give back to the community and help it grow!
* Help with unanswered forum questions and issues
* Register or login to share your knowledge at your own blog

HH's picture
HH (not verified)
Re: Nightly batch processes????

5000 employees taking 3 hours using CI. Boy, that will kill us if we use CI then... We can easily touch that number on a gcertains days of the week.

have you tried isolating the process schduler/ appserver where you run from the appserves/process schedulers that users use ? Just a thought. Your PS admin might be able to provide some inputs.

Guest's picture
Guest (not verified)
Re: Nightly batch processes????

Don't run the batch processes at all..No worries

Guest's picture
Guest (not verified)
Re: Nightly batch processes????

Thats a good idea.. but very hard to implement..

Dhanya's picture
Dhanya (not verified)
Re: Nightly batch processes????

Hi,
Group the countries based on time zones and schedule them at their night time.

Ashar's picture
User offline. Last seen 3 years 33 weeks ago. Offline
Joined: 03/26/2008
Posts: 227
Re: Nightly batch processes????

Hi Dhanya,

Thanks for your comments.

The batch program which i was talking about is processing the employees one by one and updates teh Job data via CI_JOB_DATA but still it locks complete Job Data component.

Anyway we are using the approach u mentioned above for some other processes.

Peace,

Ashar

Zooz's picture
User offline. Last seen 11 years 6 weeks ago. Offline
Joined: 06/27/2008
Posts: 10
Re: Nightly batch processes????

Maybe you can implement a solution where you have 2 instances of the same table that you want to update. When the process runs it locks one of the tables which is being updated, the users can still access the other instance of the table, but it's still not up to date. This way they are not completely shut out and they can still access the data as of the day before.

Once the update on the other table is complete, you roll it out and it becomes available for the users.

Just a thought !

Ashar's picture
User offline. Last seen 3 years 33 weeks ago. Offline
Joined: 03/26/2008
Posts: 227
Re: Nightly batch processes????

This is similar to what i mentioned in my post as "Data Caching"

Guest's picture
Guest
full support tankini swimwear cde

Plus Size Swimwear | Swimsuits for Real Women
swimsuits1 com]Show more!

miraclesuit skirtinidd cup size bikinivip swimwearswimsuits conservativewhat size swimsuitsexy in bathing suittankinis and boy shortsswimsuit for ddclearance swimming suitspolka dot bikini swimsuitladys swim wearbikinis older womend cup underwire tankini topsplus size underwireswimwear with bust support
swimsuits1 com/Aquabelle-Extended-Inseam-Bike-Short > bike shorts women plus size
swimsuits1 com/Tropiculture-Noir-CutOut-One-Piece > cutout one piece swimwear
swimsuits1 com/Underwire-Swimsuits > underwire bathing
swimsuits1 com/Plus-Size-Cover-Ups-Swimwear-Cover-Ups-D > sassa swimsuits philippines
swimsuits1 com/Core-Slimming-Swimwear-Design-Fit-Finder > swimsuits slimming
swimsuits1 com/ > large wemon
swimsuits1 com/Plus-Size-Bikinis > all bikinis
swimsuits1 com/Clearance-Swimwear-Sale > womens swimsuits on sale
swimsuits1 com/Underwire-Swimsuits > womens bathing suits by cup size
swimsuits1 com/Bikinis-Swimwear-D > all bikini
swimsuits1 com/help/internationalShopping php > german swimsuits
swimsuits1 com/Plus-Size-Clearance-Swimwear-Sale > vintage bikini cheap
swimsuits1 com/Plus-Size-Swimwear > plussize
swimsuits1 com/Tankinis-Swimwear-D > strapless swimsuits for women
swimsuits1 com/Plus-Size-Bikinis > swimsuits plus size two piece