Thermo-Key: Human Region Segmentation from Video Using Thermal Information

Japanese version is here

Contents

What is Thermo-Key?

Introduction

Image segmentation is a method of dividing an image into several regions.

Fig1. Thermo-Key
Human region segmentation is very important for a lot of applications.
  • Superimposing onto a synthetic image
  • Weather forecast
    • Image communication
    • face to face communication
    • protecting personal privacy
  • Recognizing facial expressions
  • Detecting an intruder
The goal of this project is human region segmentation from video in real time. We don�ft want to set up a special environment. We don�ft want to restrict the person to be segmented in a particular position.

These availability and interactivity will enhance the applicability of the human region segmentation.
  • Entertainment
  • Virtual reality and augmented reality
  • Outdoor applications
  • Presentation

Related Works

  • Chroma-keying: A blue (or green) screen is utilized as a well-controlled background including sophisticated lighting condition. This is a strong restriction of this method. As you know, it is widely used especially in broad casting. It is well designed for human skin color (or blue eyes).
  • Background subtraction: The color values of pixels are compared between the current image and the previously-captured background image. Such objects those have the same color as the background cannot be segmented.
  • Depth-keying: The distance between a capturing system and objects are measured or estimated. This is not always perfect. All the objects placed in particular distance region will be segmented. This could be either merits and demerits.

Our proposal

Assumption
  • The human body has higher temperature than the surroundings.
    • Warm objects will be segmented independent of colors and distance.
    • Warm environments and cold clothes would be bad for our assumption. For this problem, we could implement a kind of hybrid method that utilizes thermal and color information.
Good News
  • By capturing infrared rays, we can measure the temperature distribution in a scene.
  • This is a passive method, and could be robust in comparison with the depth measurement used in the depth-keying.
  • Thermal vision camera is already commercially available from $10,000, and widely used for medical purposes. Ours is about $40,000.
Thermo-Key
  • A thermal vision camera is used for capturing the temperature distribution.
  • Thermal data is the key for segmenting human region from a color image.
  • This method can be specialized for human region, since we know the approximate temperature of human body.
Table1: Comparison of related works
MethodCategoryRestrictionTarget
Chroma-Key(Ultimatte)Active(lighting)Blue screensMainly human
Background SubtractionPassiveStatic BackgroundAny objects
Depth-KeyActive is betterStanding positionAny objects
Thermo-KeyPassiveThe person must be aliveMainly human

Implementation

A set of a color image and a thermal image should be an identical scene. This means that the same scene should be captured from the same viewpoint at the same time.For this purpose, we have implemented a capturing system using a infrared-ray mirror.

Fig.2 Camera system

Fig.3 System flow
Pixelmap technology controls the conversion from thermal to alpha. This makes our system robust for several temperature conditions. We can control the contrast and transparency of synthetic image interactively.

Fig.4 Pixmap

Innovative Visual Effects

The following 7 visual effects are proposed and implemented here.
  • Temperature-Based Color Control
  • Shadow-Like Effect Creating the Presence of Somebody
  • Interaction Using Cold Objects
  • Interaction Using Warm Objects
  • Opaque for Visible Rays while Transparent for Infrared Rays
  • Superimposing on the Same Scene
  • Taking a Set of Yourself Snapshots

Publication

  • Yoshihiro Kawahara, Takeshi Naemura, Hiroshi Harashima:
    ``See-Through Video Avatar Using Thermal Vision Camera'', Proceedings of the 2000 IEICE General Conference, A-16-22 (2000,3) in Japanese
  • Yoshihiro Kawahara, Takeshi Naemura, Hiroshi Harashima:
    ``See-Through Video Avatar - concept and thermal vision based system -'', Proceedings of VRSJ 2000, pp. 333 -- 336 (2000,9) in Japanese
  • Takeshi Matsushita, Takuya Nitta, Takeshi Naemura, Hiroshi Harashima:
    ``See-Through Video Avatar - concept and thermal vision based system - Improvements and applications of See-Through Video Avatar'', Proceedings of the 2001 IEICE General Conference, A-16-17 (2001,3) in Japanese
  • Yoshihiro Kawahara, Takeshi Matsushita, Takuya Nitta, Takeshi Naemura, Hiroshi Harashima:
    ``Concept of See-Through Video Avatar and its Thermal Vision Based Implementation'', The Journal of the Institute of Image Information and Television Engineers, 55, 5, pp. 746 -- 750 (2001,5) in Japanese
  • Takeshi Matsushita, Takuya Nitta, Takeshi Naemura, Hiroshi Harashima:
    ``See-Throught Video Avatar for Shared Space Communication'', 3-D conference 2001, pp. 29 -- 32 (2001.7) in Japanese
  • Kazutaka Yasuda, Kaoru Sugita, Keita Ushida, Takeshi Naemura, Hiroshi Harashima:
    ``A Communication and Presentation Support System Using See-Through Video Avatars'', Proceedings of Human Interface Society symposium 2002, 3144, pp. 589 -- 592 (2002.9) in Japanese
  • Kazutaka Yasuda, Kaoru Sugita, Keita Ushida, Takeshi Naemura, Hiroshi Harashima:
    ``A Mutual Communication-Support Using See-through Video Avatars'', Proceedings of VRSJ 2002, pp. 495 -- 498 (2002.9) in Japanese
  • Kazutaka Yasuda, Takeshi Naemura, Hiroshi Harashima:
    ``Thermo-key: Human Region Segmentation from Video Using Thermal Information'', ACM SIGGRAPH2003 Emerging Technologies, Booth No. 40, San Diego (2003.7)

Materials and Contact Information

Presentation at SIGGRAPH2003 : PDF & HTML(PowerPoint)

Contact : [email protected]

Back to top page