TY - GEN
T1 - Dixit
T2 - 2019 World Wide Web Conference, WWW 2019
AU - Hsu, Chao Chun
AU - Lin, Hsin Yu
AU - Chen, Yu Hua
AU - Huang, Ting Hao
AU - Chen, Zi Yuan
AU - Ku, Lun Wei
N1 - Funding Information:
This research is partially supported by Ministry of Science and Technology, Taiwan, under Grant no. MOST108-2634-F-001-004-, MOST108-2634-F-002-008- and MOST107-2218-E-002-009-.
Funding Information:
This research is partially supported by Ministry of Science and Technology, Taiwan, under Grant no. MOST108-2634-F-001-004-, MOST108-2634-F-002-008-and MOST107-2218-E-002-009-.
Publisher Copyright:
© 2019 IW3C2 (International World Wide Web Conference Committee), published under Creative Commons CC-BY 4.0 License.
PY - 2019/5/13
Y1 - 2019/5/13
N2 - In this paper, we introduce Dixit, an interactive visual storytelling system that the user interacts with iteratively to compose a short story for a photo sequence. The user initiates the process by uploading a sequence of photos. Dixit first extracts text terms from each photo which describe the objects (e.g., boy, bike) or actions (e.g., sleep) in the photo, and then allows the user to add new terms or remove existing terms. Dixit then generates a short story based on these terms. Behind the scenes, Dixit uses an LSTM-based model trained on image caption data and FrameNet to distill terms from each image, and utilizes a transformer decoder to compose a context-coherent story. Users change images or terms iteratively with Dixit to create the most ideal story. Dixit also allows users to manually edit and rate stories. The proposed procedure opens up possibilities for interpretable and controllable visual storytelling, allowing users to understand the story formation rationale and to intervene in the generation process.
AB - In this paper, we introduce Dixit, an interactive visual storytelling system that the user interacts with iteratively to compose a short story for a photo sequence. The user initiates the process by uploading a sequence of photos. Dixit first extracts text terms from each photo which describe the objects (e.g., boy, bike) or actions (e.g., sleep) in the photo, and then allows the user to add new terms or remove existing terms. Dixit then generates a short story based on these terms. Behind the scenes, Dixit uses an LSTM-based model trained on image caption data and FrameNet to distill terms from each image, and utilizes a transformer decoder to compose a context-coherent story. Users change images or terms iteratively with Dixit to create the most ideal story. Dixit also allows users to manually edit and rate stories. The proposed procedure opens up possibilities for interpretable and controllable visual storytelling, allowing users to understand the story formation rationale and to intervene in the generation process.
UR - http://www.scopus.com/inward/record.url?scp=85066916755&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85066916755&partnerID=8YFLogxK
U2 - 10.1145/3308558.3314131
DO - 10.1145/3308558.3314131
M3 - Conference contribution
AN - SCOPUS:85066916755
T3 - The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019
SP - 3531
EP - 3535
BT - The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019
PB - Association for Computing Machinery, Inc
Y2 - 13 May 2019 through 17 May 2019
ER -