1 Detecting Visual Situations with Convolutional Networks and Active Visual Search

2 Object Detection

3 Object Detection Hat Dog

4 Object Detection Hat Hat Person Dog Football Person

5 Object Detection Hat Hat Person Dog Football Person Person PersonTable

6 Situation Detection

7 Situation Detection Person Holding Leash Walking Dog Attached to

8 Situation Detection Facing Attached to Person 1 Person 2 Handshake

9 Situation Detection Player 1 Player 2 Facing Ping Pong Table

10 What are the difficult problems in situation detection?

11 Situation Detection

12 But...

13 But...

14 Person Dog leash attached to action holds running “Dog walking” Dog Group skateboarding walking

15 “conceptual slippage”Person Dog leash attached to action holds running Allowing “conceptual slippage” “Dog walking” Dog Group skateboarding walking

16 But...

17

18 Person Dog leash attached to action holds running “Dog walking” Dog Group skateboarding walking Tail Cat Iguana

19 But...

20

21

22 ttp://thedaemon.com/images/DARPA_Segue_Dog.jpg

23

24 http://www. k9ring. com/blog/image. axd

25

26

27 Helicopter “Dog walking” Tail Person Dog leash attached to walking action holds Dog Group Cat Car skateboarding running Driving Iguana Biking Segue-ing Horse Treadmill-ing

28 Situate: An architecture for visual situation detectionSituate’s task: Decide if a given image is an instance of / analogous to a known situation (here, “dog-walking”), by grounding situation to objects in image. If it is an instance, rate how good an instance / analogy it is.

29 Situate demo

30 Current Collaborators on the Situate ProjectPortland State University: Max Quinn Jordan Witte Anthony Rhodes Sheng Lundquist Los Alamos National Laboratory: Garrett Kenyon This work is supported by the National Science Foundation.