3 tips on how to make the best use of LinkedIn Pages 🏢  16 June 2023

Meta has shared a new generative AI paper 🤖

The blog post on Meta AI's website introduces the first AI model based on Yann LeCun’s vision for more human-like AI, called the Image Joint Embedding Predictive Architecture (I-JEPA). This model learns by creating an internal model of the outside world, comparing abstract representations of images rather than the pixels themselves.

I-JEPA has shown strong performance on multiple computer vision tasks and is more computationally efficient than other widely used computer vision models. The representations learned by I-JEPA can be used for many different applications without needing extensive fine-tuning. For instance, a 632M parameter visual transformer model was trained using 16 A100 GPUs in under 72 hours, achieving state-of-the-art performance for low-shot classification on ImageNet with only 12 labeled examples per class.

The I-JEPA model is based on the concept that humans learn a significant amount of background knowledge about the world just by passively observing it. AI researchers have tried to devise learning algorithms that capture this common sense background knowledge and then encode it into a digital representation the algorithm can access later.

The I-JEPA model is a step towards a broadly capable joint-embedding predictive architecture. It predicts missing information in an abstract representation that’s more akin to the general understanding people have. Compared to generative methods that predict in pixel/token space, I-JEPA uses abstract prediction targets for which unnecessary pixel-level details are potentially eliminated, leading the model to learn more semantic features.

The blog post also mentions that I-JEPA pretraining is computationally efficient and doesn't involve any overhead associated with applying more computationally intensive data augmentations to produce multiple views. Only one view of the image needs to be processed by the target encoder, and only the context blocks need to be processed by the context encoder.

The post concludes by stating that I-JEPA demonstrates the potential of architectures for learning competitive off-the-shelf image representations without the need for extra knowledge encoded through hand-crafted image transformations. The team is looking forward to extending the JEPA approach to other domains, like image-text paired data and video data. In the future, JEPA models could have exciting applications for tasks like video understanding. This is an important step towards applying and scaling self-supervised methods for learning a general model of the world.

facebook.com