JATRIX
Programmer, Visual designer, Director - Ryan Yuan
Developed by P5.js, P5.speech.js
A Collaborative Project
Nov 2019 - Dec 2019
Project Run-through
Summary
Intention
Jatrix seeks the boundary between humanity and computer interfaces by conducting all the interactions restricted to speaking, watching, listening, and moving, different from the traditional way of touching controlling interfaces. Audiences can feel that they are interacting with something alive, and they will further think about our relationship with AI, referring to the classic question, are they alive?
JATRIX is an individual interactive webpage project, utilizing sound recognition and pixel manipulation to deliver audiences an experience to interact with an ancient AI. The project tends to simulate an ancient artificial intelligence waiting for his master's coming back, he keeps dormant until one day the silence got disturbed, and the story begins.
Home Page
Flashing and changing Japanese characters form the letters referring to the data flow effect in Matrix. All the coordinates of these Japanese characters are scaled to locate at the correct position in various resolutions.
Introduction Page
Comparing the pixels in the current frame and the previous frame in the webcam will the AI get the outlines of the moving objects and show them on the screen. The last sentence suggests the voice key to the next page, which is, CONNECTION.
Easteregg Page
If the audience says CONNECTION before the voice key shows up on the introduction page, he/she will be led to this page. To leave this page, he/she needs to say, BREAK.
This surprise pays tribute to the game Death Stranding and one of its theme songs, Ludens, sung by Bring Me the Horizon.
Connection Page
Utilizing the same trick on the introduction page to get the outlines of moving objects, the outlines will collide with the falling characters and the AI will speak out the collided ones. This is a ritual of communication with the AI through the data flow.
Help Page
This page will pop up during the communication process to give the audiences the new voice key, BYE, to leave the page.
Easteregg Page
This page will pop up if you say, MOON, which is my professor's name. THANKS is the voice key to leave this page.
Voice Recognition
To improve the accuracy of voice recognition, I take all the words with the same pronunciation into consideration, such as BYE, BY, and BUY. Also, I find out that sayonara, BYE in Japanese, can also be recognized as equal to BYE.
Outline Capture
I compare the pixels in the current frame and the previous frame in the webcam to get the outlines of moving objects as their pixels are changing. Also, I set a threshold to determine to what extent will the pixel be considered different from itself in the current frame and the previous one.
Home Page Title
I manually locate the positions of all the Japanese characters so that they will appear at correct positions in various resolutions.
Typing Effect & Auto-read
I make an array to store all the letters and type them out one by one with a timer. Also, I line up the timing of the typing texts and the timing of the auto-reading to make the process seamless.
