brails.processors.vlm_image_classifier.clip.model module

class brails.processors.vlm_image_classifier.clip.model.AttentionPool2d(spacial_dim: int, embed_dim: int, num_heads: int, output_dim: int | None = None)

Bases: Module

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class brails.processors.vlm_image_classifier.clip.model.Bottleneck(inplanes, planes, stride=1)

Bases: Module

expansion = 4

forward(x: Tensor)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class brails.processors.vlm_image_classifier.clip.model.CLIP(embed_dim: int, image_resolution: int, vision_layers: Tuple[int, int, int, int] | int, vision_width: int, vision_patch_size: int, context_length: int, vocab_size: int, transformer_width: int, transformer_heads: int, transformer_layers: int)

Bases: Module

build_attention_mask()

property dtype

encode_image(image)

encode_text(text)

forward(image, text)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

initialize_parameters()

class brails.processors.vlm_image_classifier.clip.model.LayerNorm(normalized_shape: int | List[int] | Size, eps: float = 1e-05, elementwise_affine: bool = True, bias: bool = True, device=None, dtype=None)

Bases: LayerNorm

Subclass torch’s LayerNorm to handle fp16.

forward(x: Tensor)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class brails.processors.vlm_image_classifier.clip.model.ModifiedResNet(layers, output_dim, heads, input_resolution=224, width=64)

Bases: Module

A ResNet class that is similar to torchvision’s but contains the following changes: - There are now 3 “stem” convolutions as opposed to 1, with an average pool instead of a max pool. - Performs anti-aliasing strided convolutions, where an avgpool is prepended to convolutions with stride > 1 - The final pooling layer is a QKV attention instead of an average pool

forward(x)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class brails.processors.vlm_image_classifier.clip.model.QuickGELU(*args, **kwargs)

Bases: Module

forward(x: Tensor)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class brails.processors.vlm_image_classifier.clip.model.ResidualAttentionBlock(d_model: int, n_head: int, attn_mask: Tensor | None = None)

Bases: Module

attention(x: Tensor)

forward(x: Tensor)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class brails.processors.vlm_image_classifier.clip.model.Transformer(width: int, layers: int, heads: int, attn_mask: Tensor | None = None)

Bases: Module

forward(x: Tensor)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class brails.processors.vlm_image_classifier.clip.model.VisionTransformer(input_resolution: int, patch_size: int, width: int, layers: int, heads: int, output_dim: int)

Bases: Module

forward(x: Tensor)

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

brails.processors.vlm_image_classifier.clip.model.build_model(state_dict: dict)

brails.processors.vlm_image_classifier.clip.model.convert_weights(model: Module)

Convert applicable model parameters to fp16