💡

o·ver·ride: interrupt the action of (an automatic device), typically in order to take manual control.

Overview
setup
overrideCode
Convert vs. Convert Type
Convert Type (Order: 1)
Convert (Order: 2)
Convert Type - Final (Order: 3)
Class Documentation
Holder
Constructor
Members
ParsedName
Constructor
Members
ParsedPhone
Constructor
Members
RestService
Constructor
Members
References
Examples
Change a field
Modify an amount field
Bin a field for validation
Copy a field to another
Calculate a derived value
Drop Records
Regular Expression (RegEx) Match
Plug a date 1 year ago
Plug Dates
Parsing JSON
Get year, month, day from a date
Email Verification 1
Email Verification 2
Filename
Plug Record Id or Line Number
split DOB components
Get an original source field
Concatenate several fields together
Access Client or File Attributes
Get CreateDate from File
Caps Case a Set of Fields
Create a Banded Range Field
Reduce: Keep the Latest Record Only
Reduce: Aggregate Example

Overview

💡

Users are able to implement more complex logic using one or more code overrides. Code can be injected at the convert level, affecting on that specific convert or at the convert type level, which would apply the logic to all pipelines of that specific type.

new RuntimeOverride {

  override def setup(client: Option[Client],file: Option[File]): Option[Any] = {
      //Return a global here, run once, available as setup below
      None
  }

  override def overrideCode(holder: Holder, setup:Option[Any]): Holder = {
    
		//Custom logic here
    val state = if(holder.getFieldAs[String]("state").getOrElse("UU") == "AX"){
      "AZ"
    }
    else{
      holder.getFieldAs[String]("state").getOrElse("UU")
    }
    
    holder.upsertField("state",Some(state))

  }
}

setup

💡

Use setup to perform an operation one time, and pass the result of that operation to overrideCode. For instance, if you pull a lookup data table from an API, rather than doing it in overrideCode which would call an API many times, make the API call one time in setup, then use the resulting data in overrideCode.

overrideCode

💡

Code run in overrideCode is executed on each row, represented by a Holder. It must return an instance of a Holder object.

Convert vs. Convert Type

💡

Convert Type code is run before the code for the specific convert. Convert type code must be enabled by checking then Enabled checkbox and saving. Other than the ordering and enabled, both operate in a similar way.

Convert Type (Order: 1)

Administration → Convert Types → Click convert hyperlink → Code

Convert (Order: 2)

Files → Menu for specific file → Open in Builder → Code

Convert Type - Final (Order: 3)

Class Documentation

Holder

💡

Holder is the default container used to serialize data through the conversion process. It can be changed using code overrides. Use the copy method or upsertField to mutate an instance of a holder to return.

Constructor

new io.phase3.transform.model.Holder(lineNumber: Long = 0, parsedInput: Option[Array[String]] = None, mappedInput: Option[Map[String, Option[Any]]] = None, errors: List[TransformationError] = Nil, errorList: Option[List[String]] = None, convertConfiguration: Option[ConvertConfiguration] = None, file: Option[File] = None, client: Option[Client] = None, matchcode: Option[String] = None, fieldConfiguration: Option[FieldConfiguration] = None)

Members

val client: Option[Client]

val convertConfiguration: Option[ConvertConfiguration]

def copy(lineNumber: Long = lineNumber, parsedInput: Option[Array[String]] = parsedInput, mappedInput: Option[Map[String, Option[Any]]] = mappedInput, client: Option[Client] = client, file: Option[File] = file, errors: List[TransformationError] = errors, matchcode: Option[String] = None, errorList: Option[List[String]] = errorList, convertConfiguration: Option[ConvertConfiguration] = convertConfiguration, fieldConfiguration: Option[FieldConfiguration] = fieldConfiguration): Holder

val errorList: Option[List[String]]

val errors: List[TransformationError]

val fieldConfiguration: Option[FieldConfiguration]

val file: Option[File]

def getField(k: String): Option[Any]

def getFieldAs[T](k: String): Option[T]

def getFieldAsDateTime(k: String): Option[DateTime]

def getMatchcode(): Option[String]

val lineNumber: Long

def removeField(k: String): Holderf

def toString(): String

def upsertField(k: String, v: Option[Any]): Holder

def validField(str: String): Boolean

ParsedName

Constructor

new io.phase3.transform.model.ParsedName(prefix: Option[String] = None, firstInitial: Option[String] = None, first: Option[String] = None, nicknames: Option[String] = None, middle: Option[String] = None, last: Option[String] = None, suffix: Option[String] = None, postnomial: Option[String] = None, gender: Option[String] = None, firstNameValidation: Option[NameValidation] = None, lastNameValidation: Option[NameValidation] = None, soundEx: Option[SoundEx] = None)

Members

val first: Option[String]

val firstInitial: Option[String]

val firstNameValidation: Option[NameValidation]

def fullname(): Option[String]

val gender: Option[String]

val last: Option[String]

val middle: Option[String]

val nicknames: Option[String]

val postnomial: Option[String]

val prefix: Option[String]

val soundEx: Option[SoundEx]

val suffix: Option[String]

def toString(): String

ParsedPhone

Constructor

new io.phase3.transform.model.ParsedPhone(original: Option[String] = None, formatted: Option[String] = None, isValid: Option[Boolean] = None)

Members

val formatted: Option[String]

val isValid: Option[Boolean]

val original: Option[String]

RestService

Constructor

💡

The io.phase3.transform.service.RestService class is static.

Members

def delete(uri: String, headers: Option[Map[String, String]] = None): Option[Int] //Response Code


def get[T](uri: String, headers: Option[Map[String, String]] = None (implicit arg0: Manifest[T]): Option[T]

def post[T](uri: String, body: String, headers: Option[Map[String, String]] = None (implicit arg0: Manifest[T]): Option[T]

References

Scala Cheat Sheet

Scala REPL Online

Examples

Change a field


val state = if(holder.getFieldAs[String]("state").getOrElse("XX") == "AX"){
    Some("AZ")
}
else{
    holder.getFieldAs[String]("state")
}

holder.upsertField("state",state)

Modify an amount field

val amount = holder.getFieldAs[Double]("amount").getOrElse(0.0) * 1.64

holder.upsertField("amount", Some(amount))

Bin a field for validation

val dollar = holder.getFieldAs[String]("dollar").getOrElse("0").toInt

val myBin = dollar match {
  case 0 => Some("SM")
  case x if x < 10 => Some("MD")
  case _ => Some("LG")
}

holder.upsertField("myBin",myBin)

Copy a field to another

holder.upsertField("copy_of_field1",holder.getFieldAs[String]("field1"))

Calculate a derived value

val dollars = holder.getFieldAs[Double]("dollars")
val quantity = holder.getFieldAs[Int]("quantity")

holder.upsertField("net",Option(dollars * quantity))

Drop Records

//Ensure that drop and dropReason are added as fields of this convert type

val (drop,dropReason) = if (holder.getFieldAs[String]("item_id").isEmpty) 
  (Some(true),Some("Missing item_id")) 
else 
  (None,None)

holder.upsertField("drop",drop)
      .upsertField("dropReason",dropReason)

Regular Expression (RegEx) Match

val regex = "(^[A-Za-z\\-0-9]*)".r
  
val itemId = if(holder.getFieldAs[String]("item_desc").isDefined){
  regex.findFirstMatchIn(holder.getFieldAs[String]("item_desc").get) match {
    case Some(i) => Some(i.group(1))
    case None => None
  }
}
else{
  None
}

holder.upsertField("item_id",itemId)

Plug a date 1 year ago

holder.upsertField("transactionDate",Some(new org.joda.time.DateTime().minusYears( 1 )))

Plug Dates

import org.joda.time.DateTime

val now = new DateTime()

holder.upsertField("my_date_field", Some(now))

new org.joda.time.DateTime().withYear(1900).withDayOfYear(1).withMonthOfYear(1)

Parsing JSON

val customerRefStr:String = holder.getFieldAs[String]("var01").getOrElse("{}").replaceAll("\'","\"")

    val customerRefMap:Map[String,String] = try {
      io.phase3.common.service.JsonService.deser[Map[String,String]](customerRefStr)
    } 
    catch {
        case e: Throwable => e.printStackTrace() ; Map[String,String]("value" -> "0")
    }
  
    
    holder.upsertField("customer_id",customerRefMap.get("value"))

Get year, month, day from a date

val (year,month,day) = if(holder.getFieldAs[org.joda.time.DateTime]("sale_date").isDefined){
      
      val d = holder.getFieldAs[org.joda.time.DateTime]("sale_date").get
      
      (Some(d.getYear), Some(d.getMonthOfYear), Some(d.getDayOfMonth))
    }
    else {
      (None,None,None)
    }
    
    holder.upsertField("year",year)
          .upsertField("month",month)
          .upsertField("day",day)

Email Verification 1

			import io.phase3.transform.model.EmailValidationResponse
      import io.phase3.transform.service.EmailVerificationService

      val profile = holder.getFieldAs[ProfileData]("profile")
      val (e1:Option[Boolean],e2:Option[Boolean],e3:Option[Boolean]) = if(profile.isDefined){
        
        val email01 = if(profile.get.Email_Array.getOrElse(List[String]()).isDefinedAt(0)){
          Option(profile.get.Email_Array.getOrElse(List[String]())(0))
        }
        else{
          None
        }

        val email02 = if (profile.get.Email_Array.getOrElse(List[String]()).isDefinedAt(1)) {
          Option(profile.get.Email_Array.getOrElse(List[String]())(0))
        }
        else {
          None
        }

        val email03 = if (profile.get.Email_Array.getOrElse(List[String]()).isDefinedAt(2)) {
          Option(profile.get.Email_Array.getOrElse(List[String]())(0))
        }
        else {
          None
        }
        
        (Option(EmailVerificationService.resolve(email01).getOrElse(EmailValidationResponse()).state.getOrElse("undeliverable") == "deliverable"),
          Option(EmailVerificationService.resolve(email02).getOrElse(EmailValidationResponse()).state.getOrElse("undeliverable") == "deliverable"),
          Option(EmailVerificationService.resolve(email03).getOrElse(EmailValidationResponse()).state.getOrElse("undeliverable") == "deliverable"))
      }
      else{
        (None,None,None)
      }

      holder.upsertField("email1_validated",e1)
        .upsertField("email2_validated",e2)
        .upsertField("email3_validated",e3)

Email Verification 2

		import io.phase3.transform.model.EmailValidationResponse
    import io.phase3.transform.service.EmailVerificationService

    /** **************************************************
     * Validate the client's email
     * *************************************************** */
    val clientEmailValidated = EmailVerificationService.resolve(holder.getFieldAs[String]("email")).getOrElse(EmailValidationResponse()).state.getOrElse("undeliverable") == "deliverable"


    holder.upsertField("email_validated", Some(clientEmailValidated))

Filename

val fileName = holder.file.get.alias.getOrElse(holder.file.get.fileName.getOrElse(""))

Plug Record Id or Line Number

val recordId = holder.file.get._id
val lineNumber = holder.lineNumber

//make sure record_id and line_number are fields in your convert type    
holder.upsertField("record_id",Some(recordId))
      .upsertField("line_number",Some(lineNumber))

split DOB components

new RuntimeOverride {

override def setup(client: Option[Client],file: Option[File]): Option[Any] = {
//Return a global here, run once, available as setup below
None
}

override def overrideCode(holder: Holder, setup:Option[Any]): Holder = {

import org.joda.time.DateTime
import org.joda.time.format.DateTimeFormat

val mm = holder.getFieldAs[String]("client01")
val dd = holder.getFieldAs[String]("client02")
val yyyy = holder.getFieldAs[String]("client03")

val dob = if(mm.isDefined && dd.isDefined && yyyy.isDefined){

  try {
    val formatter = DateTimeFormat.forPattern("yyyy/MM/dd")
    val dt = formatter.parseDateTime(s"${yyyy.get}/${mm.get}/${dd.get}")
    Some(dt)
  }
  catch {
      case _: Throwable => None
  }
}
else {
  None
}

holder.upsertField("dob",dob)


}

}

Get an original source field

💡

Administration → Convert Types → Output Options → Options → Passthrough Raw Input Data must be checked

# Where: field.0, field.1, field.2, field.3
holder.upsertField("test01",holder.getSourceField(0))

Concatenate several fields together

    val field1 = holder.getFieldAs[String]("field1").getOrElse("")
    val field2 = holder.getFieldAs[String]("field2").getOrElse("")
    val field3 = holder.getFieldAs[String]("field3").getOrElse("")
    
    //Concatenate fields in different ways
    val concatField = field1.concat(field2).concat(field3)
    val concatField2 = s"$field1 x $field2 x $field3"

    holder.upsertField("field4",Some(concatField))
      .upsertField("field5",Some(concatField2))

Access Client or File Attributes

new RuntimeOverride {

  override def setup(client: Option[Client],file: Option[File]): Option[Any] = {
    //Return a global here, run once, available as setup below
    Some((client,file))
  }

  override def overrideCode(holder: Holder, setup:Option[Any]): Holder = {
  
    val client = setup.get.asInstanceOf[(Option[Client], Option[File])]._1.getOrElse(Client("None"))
    val file = setup.get.asInstanceOf[(Option[Client], Option[File])]._2.getOrElse(File("None", "None"))

    /* **************************************************
     * Setup email skip flag
     **************************************************** */
    val clientAttribute1 = client.attributes.getOrElse(Map[String, Any]()).getOrElse("client_attr_1", "").asInstanceOf[String]
    val fileAttribute1 = file.attributes.getOrElse(Map[String, Any]()).getOrElse("file_attr_1", "").asInstanceOf[String]
    val clientName = client.name
    val clientId = client.id
    val fileId = file._id

    holder.upsertField("fid", Some(fileId))
  }
}

Get CreateDate from File


import scala.util.{Failure, Success, Try}
import org.joda.time.format.ISODateTimeFormat

new RuntimeOverride {
  override def setup(client: Option[Client], file: Option[File]): Option[Any] = {
    Try {
      for {
        fileId <- file.map(_._id)
        clientId <- client.map(_.id)
        url = s"http://bettrdata.back:1337/datafile/$fileId"
        response <- RestService.get[Map[String, Any]](url)
        createDate <- response.get("createDate").collect { case s: String => s }
      } yield createDate
    } match {
      case Success(result) => result
      case Failure(e) =>
        println(s"Failed getting 'createDate' in 'def setup': ${e.getMessage}")
        None
    }
  }

  override def overrideCode(holder: Holder, setup:Option[Any]): Holder = {
    val createDate = setup.map(_.asInstanceOf[String]) // this is the string version of the date
    val dateTimeFormatter = ISODateTimeFormat.dateTime()
    val createDateAsDateTime = createDate.map(dateTimeFormatter.parseDateTime) // this is the JodaTime DateTime object version of the date
    println(createDate)
    
    holder
  }
}

Caps Case a Set of Fields

    def capsCaseTestMultipleFields(h: Holder, p: List[Any]): Holder = {
      // Add in a list of field names to do this on
      val fieldList: List[String] = List("person_first_name", "person_last_name", "person_whole_name")

      val anonCapsCaseFieldValueFunc = (h: Holder, p: List[Any], columnName: String) => {
        val nameOption: Option[String] = h.getFieldAs[String](columnName)
        if (nameOption.isDefined) {
          val name = h.getFieldAs[String](columnName).map(_.trim.toLowerCase.split("\\s+").map(_.capitalize).mkString(" "))
          h.upsertField(columnName, name)
        } else {
          h
        }
      }

      val todoLocal: List[((Holder, List[Any], String) => Holder, List[Any], String)] = fieldList.map(field => (anonCapsCaseFieldValueFunc, List(), field))

      def updateHolder(h: Holder, todo: List[((Holder, List[Any], String) => Holder, List[Any], String)]): Holder = {
        todoLocal.foldLeft(h: Holder) {
          case (currentHolder: Holder, (f, p, s)) => f(currentHolder, p, s)
        }
      }

      updateHolder(h, todoLocal)
    }

Create a Banded Range Field

    import scala.util.Try
    def tryToInt( s: String ) = Try(s.toInt).toOption
    

    val age1OptInt = if(holder.getFieldAs[String]("AGE1").isDefined) tryToInt(holder.getFieldAs[String]("AGE1").get) else None

    // Example: Replace state "AX" with "AZ", otherwise keep existing state.
    val AGE1 = if (age1OptInt.isDefined) {
      val age = age1OptInt.get
      if (age >= 18 && age <= 24) Some("18-24")
      else if (age >= 25 && age <= 34) Some("25-34")
      else if (age >= 35 && age <= 54) Some("35-54")
      else if (age >= 55 && age <= 64) Some("55-64")
      else if (age >= 65) Some("65+")
      else None
    } else {
      None
    }

    holder.upsertField("AGE1", AGE1)

Reduce: Keep the Latest Record Only

💡

Make sure a Key is checked and Flag Duplicates is selected for the Convert Type. If a codeReduce is present and enabled, drop records will be ignored and will be left up to the codeReduce. Use the DUPLICATE_RECORD error and dedupeType (”drop” or “warning”) as seen in the next example to have BettrData drop/warn and report on flagged records. But in this example, the records are forcibly removed without clear reporting.

new ReduceOverride {
  override def reduceOverride(keyRecords: (String, Iterable[Holder]),dedupeType:String): Iterable[Holder] = {
    val (key, holders) = keyRecords

    val newestFirst = holders.toList.sortBy(_.getFieldAsDateTime("date").getOrElse(new org.joda.time.DateTime()).getMillis)(Ordering.Long.reverse)

    List(newestFirst.head)
  }
}

Reduce: Aggregate Example

💡

Make sure a Key is checked and Flag Duplicates is selected for the Convert Type. If a codeReduce is present and enabled, drop records will be ignored and will be left up to the codeReduce. Use the DUPLICATE_RECORD error and dedupeType (”drop” or “warning”) as seen in the example to have BettrData drop/warn and report on flagged records.


new ReduceOverride {
  override def reduceOverride(keyRecords: (String, Iterable[Holder]),dedupeType:String): Iterable[Holder] = {
    val (key, holders) = keyRecords

    val ageSum = holders.map(_.getFieldAs[Int]("age").getOrElse(0)).sum

    holders.zipWithIndex.map({ case (h, i) =>
      if (i == 0) {
        h.upsertField("age", Some(ageSum))
      }
      else {
        h.copy(errors = h.errors ::: List(new TransformationError(ErrorType.DUPLICATE_RECORD, dedupeType)))
      }
    })
  }
}

Convert Code Override

Overview

setup

overrideCode

Convert vs. Convert Type

Convert Type (Order: 1)

Convert (Order: 2)

Convert Type - Final (Order: 3)

Class Documentation

Holder

Constructor

Members

ParsedName

Constructor

Members

ParsedPhone

Constructor

Members

RestService

Constructor

Members

References

Examples

Change a field

Modify an amount field

Bin a field for validation

Copy a field to another

Calculate a derived value

Drop Records

Regular Expression (RegEx) Match

Plug a date 1 year ago

Plug Dates

Parsing JSON

Get year, month, day from a date

Email Verification 1

Email Verification 2

Filename

Plug Record Id or Line Number

split DOB components

Get an original source field

Concatenate several fields together

Access Client or File Attributes

Get CreateDate from File

Caps Case a Set of Fields

Create a Banded Range Field

Reduce: Keep the Latest Record Only

Reduce: Aggregate Example