Five Practical Challenges in Enterprise AI / ML
  • プロダクト
    • dotDataとは?
    • AutoML 2.0とは?
    • dotDataが選ばれる理由
    • dotData Cloud
    • dotData Enterprise
    • dotData Py
    • dotData Stream
  • ソリューション
    • 業界別
      • 銀行
      • 保険
      • 製造
      • 小売
      • 製薬
      • 通信
    • 役割別
      • BI & データアナリスト
      • データサイエンティスト
      • 経営層
      • IT&ソフトウェア
    • 価値別
      • 加速
      • 民主化
      • 拡張・強化
      • 業務適用
  • ニュース関連
    • プレスリリース
    • 掲載記事
  • 会社情報
    • 会社情報
    • お問い合わせ
    • 経営陣
  • ブログ
  • USAサイト
  • プロダクト
    • dotDataとは?
    • AutoML 2.0とは?
    • dotDataが選ばれる理由
    • dotData Cloud
    • dotData Enterprise
    • dotData Py
    • dotData Stream
  • ソリューション
    • 業界別
      • 銀行
      • 保険
      • 製造
      • 小売
      • 製薬
      • 通信
    • 役割別
      • BI & データアナリスト
      • データサイエンティスト
      • 経営層
      • IT&ソフトウェア
    • 価値別
      • 加速
      • 民主化
      • 拡張・強化
      • 業務適用
  • ニュース関連
    • プレスリリース
    • 掲載記事
  • 会社情報
    • 会社情報
    • お問い合わせ
    • 経営陣
  • ブログ
  • USAサイト
お問い合わせ

  • Sachin Andhare
  • Blog
  • June 23, 2020

Five Practical Challenges in Enterprise AI / ML

According to a recent Gartner blog about analytics and BI solutions, only 20% of analytical insights will deliver business outcomes through 2022. Another article by VentureBeat AI reported that 87% of data science projects never make it into production. And a global survey by Dimensional Research concluded that 78% of their AI/ML projects stall at some stage before deployment. These results indicate an exceptionally high failure rate across analytics, data science, and machine learning projects. There are many reasons why so many projects fail to meet their business objectives. In this blog, we look at the top practical challenges that enterprise AI projects face and how you can mitigate them:

  1. Identifying business problems and appropriate use cases
    While AI is an incredibly powerful technology, it is not a panacea for every business problem. Building AI because everyone is doing it and throwing any problem at it without concrete objectives is a path to failure. AI is great at sifting through massive amounts of data, discovering patterns, and finding hidden insights that otherwise are not obvious. To get started, prioritize hard to solve, complex business problems that have clear objectives. Assemble a cross-functional team of technical and functional experts, ensure buy-in from domain experts. Finally, define success criteria and measure success with relevant key metrics.
  2. Access to high-quality data
    AI and Machine learning tools rely on data to train underlying algorithms. Access to clean, meaningful data that is representative of the problem at hand is critical for the success of AI initiatives. But, enterprise data tends to be biased, noisy, outdated, unstructured, and full of errors. Many companies lack data infrastructure or do not have enough volume or quality data. Others use antiquated error-prone manual methods for data preparation resulting in inaccurate data and ultimately wrong business decisions. A typical enterprise data architecture should include master data preparation tools designed for data cleansing, formatting, and standardization before storing the data in data lakes and data marts. Data quality, data management, and governance issues are of paramount importance given the high reliance on good quality data and if overlooked, can derail any AI and ML project.
  3. Data pipeline complexity
    Data is spread across disparate databases in different formats and you need to blend and consolidate data from disconnected systems. The challenge is how to extract data, how to clean data, and reformat data to make it ready for predictive analytics. This processed data requires further manipulation that is specific to AI/ML pipelines including additional table joining and further data prep and cleansing. The process requires data engineers to write SQL code and perform manual joins to complete the remaining tasks. This complex process of data ingestion, storage, cleansing, and transformation takes time and is a major bottleneck in scaling data science operations. Automated machine learning tools such as AutoML 2.0 platforms eliminate the complexity of the data pipeline by automating a full-cycle data science workflow. Through automation, these platforms transform the raw data into the inputs of machine learning a.k.a. feature engineering, and produce predictions by combining hundreds of or even more features.
  4. Balancing model accuracy and interpretability
    There is a trade-off between prediction accuracy and model interpretability and data scientists have to do the balancing act by selecting the appropriate modeling approach. Generally speaking, higher accuracy means complex models that are hard to interpret. Easy interpretation means using simpler models but that comes by sacrificing a little bit of accuracy. Traditional data science projects tend to adopt what is known as black-box approaches that generate minimal actionable insights resulting in a lack of accountability in the decision-making process. The solution to the transparency paradox is a new approach that involves using white-box models. White-box modeling implies generating transparent features and models that empower your AI team to execute complex projects with confidence and certainty. White-box models (WBMs) provide clear explanations of how they behave, how they produce predictions, and what variables influenced the model. WBMs are preferred in many enterprise data science use cases because of their transparent ‘inner-working’ modeling process and easily interpretable behavior. Explainability is very important in enterprise data science projects. By giving insight about how the prediction models work and the reasoning behind predictions, organizations can build trust and increase transparency. AutoML 2.0 platforms automate the trade-off between accuracy and interpretability and give users the choice to select the right approach based on the use case.
  5. Model operationalization and deployment
    ML delivers value when a data scientist exports the final model from Jupiter notebook to deploy it in production. Operationalization means that the model is running in a production environment (not a sandbox environment), connected to business applications, and making predictions using live data. This last mile deployment has been a slow, manual, and prolonged process rendering the models, and insights obsolete. It can take anywhere between 8 to 90 days to deploy a single model in production. Irrespective of the AI and ML platform used, it should provide end-points to run and control the developed pipeline, and easily integrate with other business systems using standard APIs. There are several approaches to moving models into production. You need to think through batch vs real-time prediction and take into account whether real-time prediction service is feasible in terms of cost, infrastructure, and complexity. Deployment also includes monitoring the model performance, capturing the performance degradation, and updating models as necessary. Automation makes enterprise-level, end-to-end data science operationalization possible with minimum effort and maximum impact, enabling enterprise data science and software/IT teams to operationalize complex data science projects. Every enterprise data science project should start with a plan to deploy models in production to capture value and realize AI’s potential.
Read More

Recent Posts

  • AutoMLの普及は、データサイエンティスト時代の終わりを意味するか?
  • NECとdotData、SaaS型クラウドサービス「dotDataCloud」を日本で販売開始
  • dotData、Amazon SageMakerを利用し、dotData StreamのMLOps機能を強化
  • dotData、Microsoft Azureへのデプロイをサポート、 Microsoft Azure Marketplaceにて提供開始 dotDataがAzure上で利用可能となり、 Azureユーザーのデータサイエンスおよび機械学習プロジェクトを加速
  • 日本経済新聞 – NECとテックファーム、dotData展開におけるリセラー契約を締結

Search

Recent Comments

    Archives

    • November 2020
    • October 2020
    • September 2020
    • August 2020
    • July 2020
    • June 2020
    • May 2020
    • April 2020
    • March 2020
    • February 2020
    • January 2020
    • December 2019
    • November 2019
    • October 2019
    • September 2019
    • August 2019
    • July 2019
    • June 2019
    • May 2019
    • April 2019
    • March 2019
    • February 2019
    • January 2019
    • December 2018
    • November 2018
    • October 2018
    • July 2018
    • March 2018

    Categories

    • Blog
    • Events
    • Media
    • Media-JP
    • Press Releases EN
    • Press Releases JP
    • Webinars
    • White Papers

    Meta

    • Log in
    • Entries feed
    • Comments feed
    • WordPress.org
    dotData Logo in white

    Follow us on

    About

    • プロダクト
      • dotDataとは?
      • AutoML 2.0とは?
      • dotDataが選ばれる理由
      • dotData Cloud
      • dotData Enterprise
      • dotData Py
      • dotData Stream
    • ソリューション
      • 業界別
        • 銀行
        • 保険
        • 製造
        • 小売
        • 製薬
        • 通信
      • 役割別
        • BI & データアナリスト
        • データサイエンティスト
        • 経営層
        • IT&ソフトウェア
      • 価値別
        • 加速
        • 民主化
        • 拡張・強化
        • 業務適用
    • ニュース関連
      • プレスリリース
      • 掲載記事
    • 会社情報
      • 会社情報
      • お問い合わせ
      • 経営陣
    • ブログ
    • USAサイト

    News and Events

    • プロダクト
      • dotDataとは?
      • AutoML 2.0とは?
      • dotDataが選ばれる理由
      • dotData Cloud
      • dotData Enterprise
      • dotData Py
      • dotData Stream
    • ソリューション
      • 業界別
        • 銀行
        • 保険
        • 製造
        • 小売
        • 製薬
        • 通信
      • 役割別
        • BI & データアナリスト
        • データサイエンティスト
        • 経営層
        • IT&ソフトウェア
      • 価値別
        • 加速
        • 民主化
        • 拡張・強化
        • 業務適用
    • ニュース関連
      • プレスリリース
      • 掲載記事
    • 会社情報
      • 会社情報
      • お問い合わせ
      • 経営陣
    • ブログ
    • USAサイト

    Resources

    • プロダクト
      • dotDataとは?
      • AutoML 2.0とは?
      • dotDataが選ばれる理由
      • dotData Cloud
      • dotData Enterprise
      • dotData Py
      • dotData Stream
    • ソリューション
      • 業界別
        • 銀行
        • 保険
        • 製造
        • 小売
        • 製薬
        • 通信
      • 役割別
        • BI & データアナリスト
        • データサイエンティスト
        • 経営層
        • IT&ソフトウェア
      • 価値別
        • 加速
        • 民主化
        • 拡張・強化
        • 業務適用
    • ニュース関連
      • プレスリリース
      • 掲載記事
    • 会社情報
      • 会社情報
      • お問い合わせ
      • 経営陣
    • ブログ
    • USAサイト

    • 会社概要
    • お問い合わせ
    • dotDataの経営陣