• The Idea(s)

• Timetable

• Java Media Framework
• A simple video player
• A simple video frame navigator
• An advanced video player

• Limitation and Future Upgrades

• Conclusion

• Refrences

3082 lines of codes further I finally have a program running that reflects a month worth of programming, looking through API’s and testing code. Hoping that at the same time it hides the frustration that all of this also brought (is a program ever bug-free?).

As with almost all things people start, or at least, I feel they should; they start optimistic. All the time in the world to master this Java Media Framework package thing! About 3.5 weeks later and some illusions less, you update your documentary and realize that 3.5 weeks was not as much as you actually thought. Nevertheless I must say that I really enjoyed coding this program, doing the imaging, creating the test movies and fiddle around with sound software.

I’m happy I took up a bachelor project in the multimedia and culture corner, considering that after my bachelor I will continue studying digital graphic design and interactive media. The manipulation of video and audio files will undoubtedly be a part of this and I feel programming this program as my bachelor project was the first small step in a, for me, important direction.

I’d like to thank Dhr. Eliens for giving me this opportunity.

After the first orientation, I wanted to create a Media Library, containing the basic 3 media types, images, sound and movies. I wanted an alternative way to navigate through stored media, so an experimental design came to mind. First thoughts went out to a "zoom browser", in my mind it would display the stored media in a selected directory. Subdirectories in the directory would be linked to the base directory and displayed with contents (or partially) at the same time (Figure 1)

Zooming to a specific directory would enlarge the detail of that directory, displaying more of the media inside it. Giving, in my opinion, a good overview while navigating through media libraries/directories. I started coding the needed Swing components for loading directories, detecting its contents (with file type filters) running into memory problems with loading big directories of large images, since they are all completely loaded. The way out here, would be to load small previews of the images, before implementing this - note that the displaying of one directory would look quite similar to Windows Explorer in Thumbnails mode - coding this part took me quite a while and when I wanted to have the "first frame preview" of video files (avi/mpg) I first bumped into the Java Media Framework. Realizing that this actually was the needed-to-explore part, I dove into the Sample Programs, API’s and Tutorials. I found out that it wasn’t as related to Swing as initially thought.

I dove into the Java Media Framework and after another 3 weeks of hard work, I can present the result as the final compiled program together with this document.

• Orientation, Reading related material of mainly Swing, Creating Ideas, Outlining thoughts, Doing Tutorials, Swing Interfacing.

• More Swing Interfacing, Digging deeper into Java Media Framework, Creating first players and then manipulating FramePositioningControls, Creating custom GUI player, catching and redirecting mouseClicks in the custom interface.

• Continuing custom GUI player, Making of sample video files, Live drawing of the FPC controls, ripping the video stream, encoding effects into the video stream.

• Real time adding of (an array of) effects into video stream with a Swing-List (that one was tricky), incorporating most of the code into one program, chasing bugs, updating/clearifying the documentation.

A Simple Video Player

Implementing File:

Executing the File:

"The Java Media Framework API (JMF) enables audio, video and other time-based media to be added to Java applications and applets." [1] Java Media players support a six-state model based on the two fundamental states Stopped and Started. [2]

A video file player is created by the statement:

Executing this statement will cause the player to be in the Unrealized state. Before being able to play the file, we need to go through all the transitions in Figure 2. To handle the states we need a Controller. This is being done by adding a ControllerListener to the player:

A ControllerListerner is an interface forcing the implementing class to have the function controllerUpdate(ControllerEvent controllerEvent). This is the function that will catch the states of the player. Now we can execute:

In the JMF API we find this explanation of the method realize():

The RealizeCompleteEvent is caught by the ControllerListener, which can then give the instruction to prefetch the data in the giving url:

In the JMF API we find this explanation of the method prefetch():

After this the player is ready loading the file and can be started:

player.start();

The default controls in the bottom of the player are simply grabbed from the player (they are part of the package) by executing:

And adding them to the Panel

A Simple Video Frame Navigator

Implementing File:

Executing the File:

The Video Seeker doesn’t "play" the video file, but in stead the video can be accessed per frame. Moving forward or backward 1 frame or going to a Random Frame. It follows the same first steps as the Simple Video Player.

This time after the realize() function, it has the Program actively waiting for the player to finish realizing, so it can make sure that statements afterwards needing the playerinfo get the correct data. The active waiting is done by:

It then can safely get the FramePositioningControl from the player and convert the duration-time of the file to frames:

It then prefetches and actively waits in a similar fashion as above:

All the data is now ready for playing and we have a control fpc that enables us to seek frames, by the following commands:

The Advanced Video Player

Implementing Files:

Executing the File:

or directly:

• Start – Program: AdvancedVideoPlayer

The Advanced Video Player is more complex. It is different from a default Player in a number of ways. Firstly, the most noticeable (visual) difference with the default Player is it’s Custom Control GUI. The GUI is a picture implemented by a ControlBarPictureInterface, which specifies the filename of the picture and where the Controlcoordinates are located inside the picture. This way, it would be easy to create a different picture and just have it implement the ControlBarPictureInterface in Java (take a look at WireFrameLayout.java to see the implementation of an interface). Having a MouseListener catching the coordinates of clicks in the picture, and comparing them to the coordinates specified in the ControlBarPictureInterface-implementation will give the actions needed to execute. This means you are free to make any GUI you want yourself.

The Advanced Video Player offers the ability to render effects over the video stream. In order to do this, there is an essential difference with the Simple Player. In stead of creating a Player out of the selected video file, it creates a Processor:

Processor processor = Manager.createProcessor(new MediaLocator(url));

The Processor Interface actually extends the Player Interface. In the JMF API we find this explanation why this is needed:

Processor extends the state transition cycle of a Player by adding the Configuring and Configured states. The purpose of these additional states is to further refine the realizing process. The realizing step is essentially split into two phases:

• Source information gathering - the input DataSource is queried and the input media stream is parsed to get the format information for the tracks in the stream.
• Construction - the internals of the Processor are constructed to handle the input media stream.

Between these two steps, you can program the Processor to perform specific processing on its media stream.

So we get the new State Transition Diagram:

So we configure the processor:

player.configure();

waitForState(player.Configured)

Now, it is worth noting that a Processor is not just there to play the data, it can take its input (the video-file) and create any content output. We want to use it as a player in the end. So we say this by executing:

(By giving null as a parameter, we say we want to use it as a player. The function setContentDescriptor actually takes a ContentDiscriptor-Object as it’s parameter)

In order to render effects over the video, we need to grab the VideoTrack from the videofile. A videofile can contain multiple tracks (video/audio) – so we need to get the tracks and find the video-track:

In order to render Effects over the videoTrack, we should have an array of Codecs. A Codec is a media processing unit that accepts a Buffer object as its input, performs some processing on the input data, and then puts the result in an output Buffer object. The Codec array is initially empty, but can only be filled by the EffectPanel, we’ll get to that later. Applying the (empty) Codec array to the videoTrack is done by the following 2 statements:

Codec effectChain[] = { };

After this the processor is ready to be realized and prefetched. Note that processor.realize() is not being executed explicitly, just specifying processor.prefetch() is enough, because the processor can not skip any of the states in Figure 3, so prefetch() will also realize it before it prefetches.

In order to render an effect over the VideoTrack, the Window implements the AcceptCodecs-Interface, it just says the window should implement the function:

This way the EffectPanel can call this function to reload the Processor with the new Codec Array. I was hoping to do this real-time, so with no need to restart the video-file, but this appeared not to be possible; the processor must be in the Configured state in able for it to execute the videoTrack.setCodecChain(effectChain) function and since a started or stopped processor is in respectively, the Started or Prefetched state, we have an impossibility.

The EffectPanel is simple, it has two lists; one with the Available Effects (defined in the InstalledEffects.java file) and one with the currently list of running effects. Whenever the user adds an effect, by clicking the Add-Button, it updates the Running Effects-list, add the newly added Effect to the Codecchain and finally sends this updated Codecchain to the Processor, so it can restart the video-file with its new chain.

• Unimplemented is the ability to select a graphical user interface for the player from some kind of list. So it won’t be necessary to recompile the code to change the GUI of the Advanced Player.

• The ability to auto-detect effects in a menu and update the AvailableEffectList in the EffectWindow itself is not implemented.

• The Running Effects can only be removed in the order they were added, I didn’t have time to program the re-arranging of the Codec-Array when the user would want to remove an effect that was added in the middle.

• The zoombrowser is long from completed, but feels like a real challenge, although the actual zooming could be very limited by Swing.

• The option to select multiple images in the browser and send them to a processor could be implemented to compose a video this way.

• Saving a movie or an effect over a movie to disk seems very interesting (storing images and codecs) - ContentDescriptors will probably be the way out for this.

• Compressing a videostream (saving the frames in efficient ways) seems implementable: My guess is that an avi encoded with let’s say a DivX codec, could be decoded by loading the file in a Processor, get the TrackControls, find the VideoTrack, specify a Codec over this VideoTrack, so it will be the Input buffer for the Codec, then apply the DivX decodation algorithm to each individual frame and put them in Output buffer.

This list could go on for ages, but these were the functionalities that had already crossed my mind when writing the final documentation.

Although not explored to the fullest, the Java Media Framework feels like a solid package of tools. It’s the first package for Java that enables the programmer to play and manipulate time-based-media. Everything in the package makes sense, the build-up, state-flow diagrams, implementation interfaces, etc. However, the problem with the JMF, as I run into more often with Java, is that it feels slow. Faster response times would help make the programmed code feel more responsive;we all know that manipulating time-based-media is a CPU intense process, but I can’t help feeling that it could run a lot smoother. Although I can’t bring any scientific proof for this, only experience with other programs could support this assumption in a small way.

Nevertheless I can recommend the Java Media Framework for this kind of programming, the transparency of the package is great and knowing that it’s possible to build the delivered program in such a short number of time, I know that the Java Media Framework will satisfy almost all of the basic time-based-media needs.

http://java.sun.com/products/java-media/jmf/

http://www.javaworld.com/javaworld/jw-04-1997/jw-04-jmf.html

http://java.sun.com/products/java-media/jmf/2.1.1/apidocs/