Skip to content
Vladimir Mandic edited this page Aug 6, 2021 · 49 revisions

Human Library

AI-powered 3D Face Detection & Rotation Tracking, Face Description & Recognition,
Body Pose Tracking, 3D Hand & Finger Tracking, Iris Analysis,
Age & Gender & Emotion Prediction, Gesture Recognition


JavaScript module using TensorFlow/JS Machine Learning library

  • Browser:
    Compatible with both desktop and mobile platforms
    Compatible with CPU, WebGL, WASM backends
    Compatible with WebWorker execution
  • NodeJS:
    Compatible with both software tfjs-node and
    GPU accelerated backends tfjs-node-gpu using CUDA libraries

Check out Live Demo for processing of live WebCam video or static images


Demos

Project pages

Wiki pages

Additional notes


See issues and discussions for list of known limitations and planned enhancements

Suggestions are welcome!



Inputs

Human library can process all known input types:

  • Image, ImageData, ImageBitmap, Canvas, OffscreenCanvas, Tensor,
  • HTMLImageElement, HTMLCanvasElement, HTMLVideoElement, HTMLMediaElement

Additionally, HTMLVideoElement, HTMLMediaElement can be a standard <video> tag that links to:

  • WebCam on user's system
  • Any supported video type
    For example: .mp4, .avi, etc.
  • Additional video types supported via HTML5 Media Source Extensions
    Live streaming examples:
    • HLS (HTTP Live Streaming) using hls.js
    • DASH (Dynamic Adaptive Streaming over HTTP) using dash.js
  • WebRTC media track




Default models

Default models in Human library are:

  • Face Detection: MediaPipe BlazeFace - Back variation
  • Face Mesh: MediaPipe FaceMesh
  • Face Iris Analysis: MediaPipe Iris
  • Face Description: HSE FaceRes
  • Emotion Detection: Oarriaga Emotion
  • Body Analysis: MoveNet - Lightning variation
  • Hand Analysis: MediaPipe Hands
  • Body Segmentation: Google Selfie
  • Object Detection: CenterNet
  • Body Segmentation: Google Selfie

Note that alternative models are provided and can be enabled via configuration
For example, PoseNet model can be switched for BlazePose model depending on the use case

For more info, see Configuration Details and List of Models




Human library is written in TypeScript 4.2
Conforming to JavaScript ECMAScript version 2020 standard
Build target is JavaScript EMCAScript version 2018


For details see Wiki Pages
and API Specification


Clone this wiki locally