First, if you are just looking for a script that will do this for you, jump to the code and take what you need. However, this post is a full tutorial that explains not only how it works, but the methodology behind getting it working. If you are impatient, the code block is halfway down the page. It is BSD licensed so you can use it however you like, just remember to give credit where its due.

Recently I have had need to set up an IP camera to perform security duties in a pawn shop. The software that comes with the camera only works on Windows, and the built in viewer on the camera itself only works through ActiveX applets in Internet Explorer. However, at this particular pawn shop, they use GNU/Linux based computers running Ubuntu for most general uses.

This posed a bit of a sticky problem. How do I record the stream off the camera when there is no Windows computer to do it? The shop is not willing to pay the cost of installing a Windows server just to record video. But even if that did work, the end result would be next to useless. They’d only be able to use the proprietary software that came with the camera to record and playback. What about being able to see it on their Linux based desktops? Or what about being able to view it on their cellphones?

Hmmm…. Steve must think long about this.

Playing with the camera’s web interface turns up a few things of value.

1) The camera can take snapshots as quickly as 1 image every 1 second.
2) The camera supports uploading to FTP servers.

Now, I don’t know about you guys, but even though 1 second may be a long time, but its short enough that you can capture pretty much everything that happens quite clearly.

So on to the Linux server goes an FTP daemon and a new user account to receive inbound FTP connections from the camera. In this account’s home directory is a directory called “recordings” that is specifically to receive the images uploaded by the camera.

Great. Now we have that working. But we’re getting 1 image a second. That 3600 individual images an hour. All ending up in the same directory. Fortunately the camera allows us to add the time stamp to the filename of the image. So in this case we would end up with a filename something like <cam-name>_<YYYYMMDDHHMMSS>.jpg. A pig to say the least, but useful. Any time we have a set format like this, we can use it :-)

So first we need to get these images under control.

Because the files are time stamped in their filename, we can actually use the filename to filter the files out of the /recordings/ directory.

Two things I wanted out of this.

I wanted the files separated by date, but also by time. Its not difficult to keep track of 3600 files, but 360000 in a few short days would be difficult rather rapidly. So we need a plan on how to archive the files that makes them easily accessible without being too difficult to manage. The time stamps make this really easily. If we move the images to an directory folder every hour, we can create that directory structure really simply.

At this point, my directory structure now looks something like this.

/
–/recordings/
–/archives/
—-/YYYYMMDD/
——/HH/

This is simple. There are 365 days in a year, but we’re not likely to store the data that long anyway. Chances are when something happens we’re going to know relatively quickly. A few months of archives is fine.

The second thing I wanted out of this was to get these images into a format that is really easy to review. Scrolling through 3600 images is a punk. Especially when that 3600 images only accounts for a single hour and there are 24 hours in a day. Have you ever tried to find a needle in a haystack? Looking for an event in 86,400 images is much the same. Time stamped or not, thats still a lot of images to review.

The easiest way to do such a thing would be to convert the images into a movies.

This also fits in with the desire to view the footage on a cellphone. Most cellphones would crap themselves over 3600 images, let alone 86,400 a day. So converting those images to something the phone can handle a little better makes a lot of sense.

Disk space is also an issue. I don’t want to make movies that will consume more space than the images they’re made from. A 1 for 1 parity is fine, but I don’t want to double my storage requirements in a given day. With the cameras current settings, its using about 60MB an hour just for the images as it is. Thats not too bad though because I had expected about 80MB to 100MB per hour. So we’re already ahead of my expectations.

Taking all this into consideration, I settled on MP4 video files as the output. Nearly all cellphones support them now days without any extra software. Every computer can play them, regardless of OS. (If you’re really struggling to figure it out, you can just install VLC [1] and be done with it.) So it makes for a very useful cross platform video container. Size shouldn’t be too much of an issue. MP4 is pretty good with its compression, and I’m not adding any audio. If I keep the bit rate to the same resolution as the original images, things should work out well.

So lets recap.

  • Every hour 3600 images are uploaded by the camera via FTP to a /recordings/ directory.
  • Every hour we need to move those folders to a directory at /archives/YYYYMMDD/HH/
  • Every hour we need to take those same images and convert them to an MP4 movie that can be easily transported to other platforms, including cellphones

So lets look at the final code I came up with, and then we’ll break it down into its pieces.

#!/bin/bash

# Simple Camera Image Archiver by Stephen Cropp of skcservices.com
# Copyright (C) 2011 by Stephen Cropp

# This script is released under the terms of the BSD license.
# You can find a summary and text of the license terms at the following website
# http://creativecommons.org/licenses/BSD/

# THIS SOFTWARE IS PROVIDED BY STEPHEN CROPP "AS IS" AND ANY EXPRESS OR IMPLIED
# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
# EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ldate=$(date +%Y%m%d -d "1 hour ago")
lhour=$(date +%H -d "1 hour ago")

archive=/home/camera/archives
recordings=/home/camera/recordings

mkdir -p $archive/$ldate/$lhour/
mv $recordings/cam1_$ldate$lhour*.jpg $archive/$ldate/$lhour/

x=1;
for f in $(ls -r -t $archive/$ldate/$lhour/*jpg); do
        counter=$(printf %04d $x);
        cp "$f" /home/camera/tmp/cam1_"$counter".jpg;
        x=$(($x+1));
done
ffmpeg -y -f image2 -r 4 -i /home/camera/tmp/cam1_%04d.jpg $archive/$ldate/$ldate.$lhour.00-59.mp4
rm /home/camera/tmp/cam1_*.jpg

This script gets run every hour via crontab. You can use whatever scheduler you like, I just happen to be a crontab kind of guy. To account for any slight time decrepancies between the camera and the server, I generally set the script to run 5 minutes after the hour.

For the most part, it is really really simple. We create 4 variables. These are lines 21 through 25 above.

$ldate
Contains the date one hour ago.
$lhour
Contains the hour one hour ago in 24hr format
$archive
Contains the path to the archives. Where we want the images copied to.
$recordings
Contains the path to where the camera is uploading the individual JPG images.

On line 27 the first thing we need to do is make sure the folder for the images exists. If it doesn’t exist, make it. If it does exist, make it anyway just to be on the safe side.

Because we know that all the files are time stamped in the same way, we can build the filename from our previous variables $ldate and $lhour.

Line 28 basically says take all the images from the recordings directory that were created in the previous hour and move them to the directory created for that hour on line 27. Simple and straight forward without too much effort.

On line 30 is where the stuff most of you will want begins. Lines 30 through 35 were copied from the ffmpeg FAQ page on their site. [2] I’ve broken it up into multiple lines to make it easier to read and follow.

ffmpeg can convert images to video, on the condition those images are all numbered sequentially and in the same format with the same size. Getting the files numbered sequentially is an issue. Initially I thought I could just use the filenames to do it. Unfortunately thats not quite going to work. The files use a date/time to as the name. So while they are sequential, they jump in number every 60 seconds. ffmpeg can’t cope with that without lots of effort. Every file has to be numbered sequentially, properly.

First lets create a counter variable called $x, on line 30 starting at 1.

Then on line 31 we’ll get a list of all the files in the archives directory, sorted by their modified time (the time uploaded by the camera) and then store that in a variable we’ll call $f.

On line 32 we create another counter called $counter. This counter will take the current value of $x and reformat it to the size we need it to be. In our case, we’re working with 3600 files, so we need to have our filenames numbered 0001 through to 3600. The %04d basically tells printf to convert $x into a 4 digit number with padding 0′s.

On line 33 we take the next filename we have stored in $f and copy it to our temp directory with a new name sequentially numbered from 0001 to 3600.

Line 34 then increments the $x variable for the next file.

We keep repeating lines 32 through 34 until we have done it for all 3600 files in our $archives directory.

Once thats done, we run ffmpeg to convert our sequentially numbered images into an MP4 movie. Lets line 36 down a little bit as this is the one that most people seem to want.

ffmpeg -y -f image2 -r 4 -i /home/camera/tmp/cam1_%04d.jpg $archive/$ldate/$ldate.$lhour.00-59.mp4

-y
tells the ffmpeg command to overwrite any existing file. As this is running via crontab, we can’t have it sitting there forever asking us whether we want to overwrite or not.
-f image2
tells ffmpeg that the input format is going to be images.
-r 4
sets the output frame rate to 4 frames per second. This allows us to take 60 minutes of images and compress them into 15 minutes of video that are still easy to follow. It also makes it trivial to get individual frames from the resulting video.
-i /home/camera/tmp/cam1_%04d.jpg
tells ffmpeg that the input files are in that location. the cam1_%04d.jpg tells it to look for cam1_0001.jpg through to cam1_9999.jpg. As our files should only go up to 3600, we’re well within the limitations here.
$archive/$ldate/$ldate.$lhour.00-59.mp4
is our output filename. We’re putting it in the archive directory for that day and specifying the hour as part of the name, followed by the minutes of the hour that video covers. As all these videos are covering the full hour, I just added 00-59 manually. More on that later.

The end result of all this is that in $archives/$ldate/ I end up with 24 MP4 videos, each 15 minutes long but containing one hours worth of security camera footage at 4 frames per second.

Now the kicker. By keeping the settings simple, ffmpeg defaults to using the same resolution as the source images and automatically chooses the best fit bitrate. The upshot of all that is I end up with a video at the same resolution as the original files, but significantly smaller. We go from around 60MB for 3600 images, to 25MB for the video. This leaves us inside the 80-100MB of storage per hour I had already provisioned. But even more importantly, it means when storage does become an issue in the future, the videos retain all the information of the images. Thus the images can either be compressed and archived off in big tarballs to another storage location. Or they can be deleted entirely and just retain the videos.

Storage is very cheap at the moment. 2TB of storage is almost pocket change now days. So plugging in an external drive or a NAS and archiving the images to that instead of keeping them on the local machine is very simple and a viable long term archival solution if necessary.

Now, say you wanted to create a video of a specific time period.

Because the image file names contain the full date, hour, minute and second they were captured, you can create a video of any time period you want quite easily. As such, I also created another script based on the one above that takes several command line parameters. Essentially you specify the start date/time and the finish date/time and it will take all those images that make up that time period and convert them into a single movie. Very simple to achieve and not really much different from what you see above. Just a little bit of extra logic to work out the times entered and what directories the images need to be copied from.

The next step to this is to add a web interface to it all. I want to allow the client to be able to load a webpage and see exactly what was happening in their shop at any moment simply by selecting a time frame and watching the video in their browser. Because I’ve used MP4 as the container, I wouldn’t need to reconvert any of the videos to something like FLV or WebM just to view it. But I can offer those as options.

The other major advantage to all of this is that everything is portable. JPG images are supported by everything. MP4 videos are supported by nearly every major cellphone provider out there, and every OS has software that can be used to play them. Its also true that if MP4 later turns out to be a bit of an issue, its very easy to convert the videos to using OGG or WebM or something else entirely if necessary. ffmpeg is by far the swiss army knife of media conversions.

Now a lot of people will probably ask why I made such a long post about something that is essentially only a few lines long. In reality, I spent a long time browsing around the web trying to find information that would be useful for converting these images and trying to figure out a plan of attack. I found lots of people asking for help on how to do this, but very little useful information that actually explains it. Whats the point in knowing how to do something if you don’t understand how it works? And thus, like all my tutorials over the years (and the few on this site) this tutorial is aimed more at helping people understand how it works, rather than just how to do it.

Let me know, is it useful?

[1] http://www.videolan.org/
[2] http://ffmpeg.org/faq.html#SEC14