<s5>Æliens</s5> -- slide(s)

learning objectives

After reading this chapter you should be able to characterize the functionality of current multimedia platforms, to describe the capabilities of GPUs, to mention the components of the Microsoft DirectX 9 SDK, and to discuss how to integrate 3D and video.

Almost 15 years ago I bought my first multimedia PC, with Windows 3.1 Media Edition. This setup included a video capture card and a 4K baud modem. It was, if I remember well, a 100 Mhz machine, with 16 Mb memory and a 100 Mb disk. At that time, expensive as it was, the best I could afford. Some 4 years later, I acquired a Sun Sparc 1 multimedia workstation, with a video capture card and 3D hardware accelerator. It allowed for programming OpenGL in C++ with the GNU gcc compiler, and I could do live video texture mapping at a frame rate of about one per second. If you consider what is common nowadays, a 3Ghz machine with powerful GPU, 1 Gb of memory, a 1.5Mb cable or ADSL connection and over 100 Gb of disk space, you realize what progress has been made over the last 10 years.

In this chapter, we will look in more detail at the capability of current multimedia platforms, and we will explore the functionality of the Microsoft DirectX 0 platform. In the final section of this chapter, I will then report about the work I did with the DirectX 9 SDK to implement the ViP system, a presentation system that merges video and 3D.

...

4 generations of GPU

Before the introduction of the GPU, there only existed very expensive specialized hardware such as the machines from SGI.
The first generation of GPU, including NVIDIA TNT2, ATI Rage and 3dfx Voodoo3, only supported rasterizing pre-transformed triangles and some limited texture operations.
The second generation of GPUs, which were introduced around 1999, included the NVIDIA GeForce 2 and ATI Radeon 7500. They allowed for both 3D vertex transformations and some lighting, conformant with OpenGL and DirectX 7.
The tird generation GPUs, including NVIDIA GeForce 3, Microsoft Xbox and ATI Radeon 8500, included both powerful vertex processing capabilities and some pixel-based configuration operations, exceeding those of OpenGL and DirectX 7.
Finally, the fourth generation of GPUs, such as the NVIDIA GeForce FX and ATI Radeon 9700, allow for both complex vertex and pixel operations.

graphics pipeline

vertex transformation -- apply world, view, projection transforms
assembly and rasterization -- combine, clip and determine pixel locations
fragment texturing and coloring -- determine pixel colors
raster operations -- update pixel values

...

A simple morphing shader in ViP, see section 4.3.

HLSL declarations


  texture tex;
  float4x4 wvp;	// World * View * Projection matrix
  
  sampler tex_sampler = sampler_state
  {
      texture = /;    
  };

vertex shader data flow


  struct vsinput {
      float4 position : POSITION; 
      float3 normal : NORMAL; 
      float2 uv : TEXCOORD0;
  };
  struct vsoutput {
      float4 position   : POSITION;   // vertex position 
      float4 color    : COLOR0;     // vertex diffuse color
      float2 uv  : TEXCOORD0;  // vertex texture coords 
  };

vertex shader


  vsoutput vs_id( vsinput vx ) {
      vsoutput vs;
    
      vs.position = mul(vx.position, wvp);
      vs.color = color;
      vs.uv = vx.uv; 
      
      return vs;    
  }

pixel shader


  struct psoutput
  {
      float4 color : COLOR0;  
  };
  
  
  psoutput ps_id( vsoutput vs ) 
  { 
      psoutput ps;
  
      ps.color = tex2D(tex_sampler, vs.uv) * vs.color;
  
      return ps;
  }

technique selection


  technique render_id
  {
      pass P0
      {          
          VertexShader = compile vs_1_1 vs_id();
          PixelShader  = compile ps_2_0 ps_id(); 
      }
  }

...

Examples of Impasto, see examples -- impasto

morphing (vertex) shader


     float3 spherePos = normalize(vx.position.xyz);
     float3 cubePos = 0.9 * vx.position.xyz;
  
     float t = frac(speed * time);
     t = smoothstep(0, 0.5, t) - smoothstep(0.5, 1, t);
  
     // find the interpolation factor
     float lrp = lerpMin + (lerpMax - lerpMin) * t;
  
     // linearly interpolate the position and normal
     vx.position.xyz = lerp(spherePos, cubePos, lrp);
     vx.normal = lerp(sphereNormal, cubeNormal, lrp);
  
     // apply the transformations
     vs.position = mul(wvp, vx.position);

coloring (pixel) shader


      float4 x = tex2D(tex_sampler, vs.uv);
      if (x.r > x.g && x.r > x.b) { x.r *= xi; x.g *= xd; x.b *= xd; }
      else  if (x.g > x.r && x.g > x.b) { x.g *= xi; x.r *= xd; x.b *= xd; }
      else  if (x.b > x.r && x.b > x.g) { x.b *= xi; x.r *= xd; x.g *= xd; }
      ps.color = x;

...

rendering of van Gogh painting with Impasto

...

Stacks and stacks of books on DirectX

DirectX

Microsoft DirectX is an advanced suite of multimedia application programming interfaces (APIs) built into Microsoft Windows; operating systems. DirectX provides a standard development platform for Windows-based PCs by enabling software developers to access specialized hardware features without having to write hardware-specific code. This technology was first introduced in 1995 and is a recognized standard for multimedia application development on the Windows platform.

DirectX 9 components

Direct3D -- for graphics, both 2D and 3D
DirectInput -- supporting a variety of input devices
DirectPlay -- for multiplayer networked games
DirectSound -- for high performance audio
DirectMusic -- to manipulate (non-linear) musical tracks
DirectShow -- for capture and playback of multimedia (video) streams

Direct3D tutorials

tutorial 1: creating a device
tutorial 2: rendering vertices
tutorial 3: using matrices
tutorial 4: creating and using lights
tutorial 5: using texture maps
tutorial 6: using meshes


  class  drumpad {
  public:
      drumpad()
      ~drumpad();
      bool    initialize( DWORD dwNumElements, HWND hwnd );
      bool    load( DWORD dwID, const TCHAR* tcszFilename );
      bool    play( DWORD dwID );
  protected:
      void    CleanUp();
      CSoundManager* m_lpSoundManager;
      CSound **      m_lpSamples;
  };


  bool drumpad::play( DWORD id ) {
      m_lpSamples[id] -> Stop();
      m_lpSamples[id] -> Reset();
      m_lpSamples[id] -> Play( 0, 0 );
      return true;
  }

multimedia challenges

volume -- multimedia streams contain large amounts of data, which must be processed very quickly.
synchronization -- audio and video must be synchronized so that it starts and stops at the same time, and plays at the same rate.
delivery -- data can come from many sources, including local files, computer networks, television broadcasts, and video cameras.
formats -- data comes in a variety of formats, such as Audio-Video Interleaved (AVI), Advanced Streaming Format (ASF), Motion Picture Experts Group (MPEG), and Digital Video (DV).
devices -- the programmer does not know in advance what hardware devices will be present on the end-user's system.

...

DepthCube, see example(s) -- 3D vision

...


3D vision	Perspectra	DepthCube

platform-dependence -- both hardware and OS
programming language -- C/C++, Java, .NET languages
functionality -- graphics, streaming media
deployment -- PC/PDA, local or networked, web deployment

Software will be the single most important force in digital entertainment over the next decade.

...

Animation in front of television news in ViP

www.virtualpoetry.tv

The ViP system enhances your party with innovative multimedia presentations.

It supports multiple webcams and digital video cameras, mixed with video and images, enhanced by 3D animations and text, in an integrated fashion.

For your party, we create a ViP presentation, with your content and special effects, to entertain your audience.

...


(a) VMR filter	(b) multiple VMRs

...

Lecture on digital dossier for Abramovic, in ViP


  class  scene  {
  public:
  	virtual int init(IDirect3DDevice9*);   // initialize scene (once)
  	virtual int compose(float time);  // compose (in the case of an animation)
  	virtual int restore(IDirect3DDevice9*);  // restore device settings
  	virtual int render(IDirect3DDevice9*  device,  IDirect3DTexture9* texture); 
  protected:
  ...
  };


  class  scene  {
  public:
  	virtual int load();   // initialize scene (once)
  	virtual int compose();  // compose (in the case of an animation)
  	virtual int restore();  // restore device settings
  	virtual int render();  // display the (sub) scene
  protected:
  ...
  };

...


installation	reality of TV news

comparative overview

	BlC	AW	D3D	HL2	SL
in-game building	-	+	+/-	-	++
avatar manipulation	+	++	+/-	+	++
artifical intelligence	+	-	+/-	+	-
server-side scripts	+	-	+/-	+	++
client-side scripts	++	-	+/-	+	-
extensibility	+	-	++	+	+/-
open source	-	-	++	-	+/-
open standards	-	-	+/-	-	+/-
interaction	+/-	+/-	++	++	+/-
graphics quality	+/-	+/-	++	++	+
built-in physics	-	-	+	++	+
object collision	-	-	++	++	+
content tool support	+/-	-	++	+	-

...

Clima Futura architecture

module(s)

climate model(s) - action script module(s)
game play interaction - event-handler per game event
video content module - video fragment(s) and interaction overlays
minigame(s) - flash module(s) with actionscript interface
Clima Futura - integration of modules 1-4, plus server-side ranking
adapted versions -- educational, commercial
multi-user version --with server-side support

interactive application(s)

realtime interaction -- providing information
content navigation -- providing view(s)

collada

document(s) -- schema, asset, library, technique, ...
geometry -- array, accessor, meshes, vertices, polygons, ...
scene(s) -- material, light, optics, camera, imager, ...
effect(s) -- shader, profiles, techniques, pass, ...
animation(s) -- sampler, channel, controller, skin, morphing, ...
physics -- collision, equation, rigid body, constraints, force, ...

...

4. multimedia platforms

What components does a multimedia platform consist of?
Discuss both hardware and software components.

concepts

technology

projects & further reading

As a project, I suggest the development of shader programs using Rendermonkey or the Cg Toolkit, or a simple game in DirectX.

You may further explore the possibilities of platform independent integration of 3D and media, by studying for example OpenML. For further reading, among the many books about DirectX, I advice [Game], [Animation] and [Audio].

the artwork

dutch light -- photographs from documentary film Dutch Light.
ViP -- screenshot, with morphing shader, see section 4.3.
impasto -- examples, see section 4.1
impasto -- after a painting of van Gogh, using Cg shaders,
3D vision, from [DEEP], see example(s) section 4.2.
idem.
photographs of DirectX and multimedia books, by the author.
DirectX -- diagram from online documentation.
ViP -- screenshot, with the news and animations.
DirectX -- diagram from online documentation.
DirectX -- diagram from online documentation.
ViP -- screenshot, featuring Abramovic.
Peter Frucht -- Reality of TV news, see section 4.3.
Clima Futura -- architecture diagram.
signs -- people, [Signs], p. 248, 249.

Æliens

2009

multimedia platforms

document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('@tmp-4.html#slide-'); } document.write('4-x-shader-tech'); if (!slidemode) document.write('.html');document.write('>');

document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('@tmp-4.html#slide-'); } document.write('secondlife-table'); if (!slidemode) document.write('.html');document.write('>');

document.write(' <a href=');if (!slidemode) { document.write('@slide-'); } else { document.write('@tmp-4.html#slide-'); } document.write('4-4-climate-components'); if (!slidemode) document.write('.html');document.write('>');

projects & further reading